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SUBJECT: Report on the Management of Developmental and Operational Test Waivers 
for Defense Systems (Report No. D-2003-105) 


We are providing this report for review and comment. This report discusses the 
overall management of waivers of developmental and operational test requirements for 
Defense systems. The Under Secretary of Defense for Acquisition, Technology, and 
Logistics did not respond to the draft report and the Director, Operational Test and 
Evaluation did not provide comments on Recommendation A.1. In preparing the final 
report, we considered comments from the Director, Operational Test and Evaluation and 
the Military Departments on a draft of this report. | 


DoD Directive 7650.3 requires that all recommendations be resolved promptly. 
As aresult of management comments, we revised and redirected Recommendation B. to 
clarify our intention. Therefore, in response to the final report, we request that the Under 
Secretary of Defense for Acquisition, Technology, and Logistics provide comments on 
Recommendations A.1., A.2., and B.; that the Director, Operational Test and Evaluation 
provide comments on Recommendation A.1. and B; and the Military Departments 
provide comments on Recommendation B. by August 19, 2003. 


If possible, please send management comments in electronic format (Adobe 
Acrobat file only) to Aud-am@dodig.osd.mil. Copies of the management comments 
must contain the actual signature of the authorizing official. We cannot accept the 
/ Signed / symbol in place of the actual signature. | 


We appreciate the courtesies extended to the audit staff. Questions should be 
directed to Mr. John E. Meling at (703) 604-9091 (DSN 664-9091) or Mr. Jack D. Snider 
at (703) 604-9087 (DSN 664-9087). See Appendix J for the report distribution. The 
team members are listed inside the back cover. 


an Sou die. 
David K. Steensma 


Deputy Assistant Inspector General 
for Auditing 
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Management of Developmental and Operational Test Waivers 
for Defense Systems 


Executive Summary 


Who Should Read This Report and Why? Policy makers, milestone decision makers, 
and testers for Defense systems should be interested in this report because it discusses the 
importance of addressing critical operational issues before a weapon system enters 
production and the need for consistent and updated waiver-and-limitation terminology and 
procedures. 


Background. This audit was conducted based on concerns that the Director, Operational 
Test and Evaluation expressed in his “Operational Test and Evaluation Annual Report, 
FY 2000,” February 2001. Specifically, the Director expressed concern about programs 
beginning operational test and evaluation without completing sufficient developmental 
test and evaluation and about the Navy’s use of test waivers. 


Results. The independent operational test and evaluation organizations within the 
Military Departments were not able to fully resolve critical operational issues for weapon 
systems during initial operational test and evaluation. Consequently, weapon systems that 
were not fully ready continued into production before critical operational issues were 
resolved, creating a potential need for costly retrofit of fielded units and an initial 
operational performance that may be less than required to defeat the expected threat. In 
addition, the Military Departments applied inconsistent waiver-and-limitation terminology 
and procedures when referring to the deferral of testing. Although the Military 
Departments have policies and procedures for waivers, only the Navy processed waivers 
to defer testing of operational requirements. Conversely, the Army and the Air Force 
cited test limitations to defer testing of operational requirements. Without consistent and 
updated waiver-and-limitation terminology and procedures for deferral of testing, the 
Military Departments’ use of test waivers is inconsistent with Office of the Secretary of 
Defense policy and affects the milestone decision makers’ ability to make fully informed 
decisions before systems enter low-rate initial production and full-rate production. 
Requiring the Military Departments to assess operational performance requirements as 
unattained in test reports when test waivers prevent a complete evaluation of operational 
requirements, requiring program managers to resolve critical operational issues before 
approval of system full-rate production, and establishing consistent guidance for test 
waivers or deferrals should bring the oversight and improvements needed to those issues. 
See the Finding section of the report for the detailed recommendations. 


Management Comments and Audit Response. We received comments from the 
Director, Operational Test and Evaluation; the Deputy Under Secretary of the Army 
(Operations Research) (the Deputy Under Secretary); the Deputy Assistant Secretary of 
the Navy (Research, Development, Test and Evaluation) (the Deputy Assistant Secretary); 
and the Director, Air Force Test and Evaluation; however, we did not receive comments 
from the Under Secretary of Defense for Acquisition, Technology, and Logistics. 


The Director, Operational Test and Evaluation did not comment on the recommendation 
concerning the Military Departments’ assessments of operational performance 
requirements; however, he partially concurred with having program managers resolve 
critical operational issues before approving systems for full-rate production. Further, he 
agreed with establishing consistent guidance for processing test waivers or deferrals. 


The Deputy Under Secretary concurred with the recommendations concerning the Military 
Departments’ assessments of operational performance requirements and the program 
managers resolving critical operational issues before approving systems for full-rate 
production, but he nonconcurred with establishing consistent guidance for processing test 
waivers or deferrals; however, he suggested a corrective action that met the intent of the 
recommendation. 


The Deputy Assistant Secretary and the Director, Air Force Test and Evaluation 
nonconcurred with the recommendations concerning the Military Departments’ 
assessments of operational performance requirements and the program managers resolving 
critical operational issues before approval of system for full-rate production. Further, they 
both nonconcurred with establishing consistent guidance for processing test waivers or 
deferrals. In addition, the Deputy Assistant Secretary stated that his comments contained 
information considered privileged in litigation, primarily under the deliberative process 
privilege and, therefore, was exempt from mandatory disclosure under the Freedom of 
Information Act. See the Finding section of the report for a discussion of the 
management comments and the Management Comments section of the report for the 
complete text of the comments. 


In response to the comments by the Deputy Under Secretary of the Army (Operations 
Research) and to obtain succinct and attainable corrective action, we revised and 
redirected the recommendation to establish consistent guidance for processing test waivers 
or deferrals. Therefore, we request that the Under Secretary of Defense for Acquisition, 
Technology, and Logistics; the Director, Operational Test and Evaluation; the Deputy 
Under Secretary of the Army (Operations Research); the Assistant Secretary of the Navy 
(Research, Development, and Acquisition); and the Director, Air Force Test and 
Evaluation comment on the revised recommendation. In addition, we request that the 
Under Secretary of Defense for Acquisition, Technology, and Logistics provide comments 
on the recommendations concerning the Military Departments’ assessments of operational 
performance requirements and the program managers resolving critical operational issues 
before approving systems for full-rate production. We also request that the Director, 
Operational Test and Evaluation provide comments on the recommendation concerning 
the Military Departments’ assessments of operational performance requirements. The 
comments on this report should be provided by August 19, 2003. 


il 
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Background 


This report discusses the overall management of waivers of developmental and 
operational test requirements for Defense systems. We conducted the audit based 
on concerns that the Director, Operational Test and Evaluation (DOT&E) 
expressed in his “Operational Test and Evaluation Annual Report, FY 2000,” 
February 2001. Specifically, the Director expressed concern about programs 
beginning operational test and evaluation without completing sufficient 
developmental test and evaluation and about the Navy’s use of test waivers. 

DoD guidance does not specifically define what a waiver from developmental and 
operational testing is. DOT&E and the Military Departments have differing 
definitions for test waivers and use different terms when referring to the deferral 
of testing. 


DOT&E Test Waiver. The Office of DOT&E defines a test waiver as deferral 
of a testing requirement, which does not eliminate the requirement for testing. 
The DOT&E definition does not include sections 2399 and 2366, title 10, United 
States Code, regarding system contractor involvement in operational test and 
evaluation and when live-fire test and evaluation is too expensive or impractical, 
respectively. 


Army Test Waiver. The Army uses a test waiver to allow testers not to conduct 
the testing specified in the approved test and evaluation master plan (TEMP); 
however, for those tests for which waivers were granted for reasons other than 
availability of data, the program manager or other appropriate official will expand 
production testing or follow-on operational testing to address those operational 
requirements not previously tested. In addition, the Army uses the terms 
“limitations” and “delays” to defer specific operational test requirements. 
Further, Army guidance requires the approved TEMP to state the test limitations 
that may affect the resolution of critical operational issues; the impact of those 
limitations, such as the effects on COIs; and the ability to formulate conclusions 
regarding operational effectiveness and operational suitability. 


Navy Test Waiver. The Navy uses the term “waiver” to mean either deferral of 
testing of an operational requirement or authorization to proceed with testing, 
even though the program has not met the requirements of Secretary of the Navy 
Instruction 5000.2B, “Implementation of Mandatory Procedures for Major and 
Non-Major Defense Acquisition Programs and Major and Non-Major Information 
Technology Acquisition Programs,” December 6, 1996. 


Air Force Test Waiver. Air Force procedures require program managers to 
process waivers to modify specific operational test requirements, and identify test 
limitations in test plans. Further, Air Force guidance requires waived items to be 
tested in subsequent operational test and evaluation or the operational 
requirements document (ORD) must be changed. 


Military Departments’ Position on DOT&E Waiver Definition. Although the 
Military Departments did not agree that test limitations and test deferrals are 
equivalent to test waivers, as defined by DOT&E, test limitations and test 
deferrals accomplish the same objective as test waivers by deferring required 


testing until later in the system acquisition process, usually after the system’s 
full-rate production decision. Further, DOT&E has indicated that test limitations 
and test deferrals are equivalent to test waivers in that they also result in the 
deferral of testing requirements. Appendix B provides a glossary of technical 
terms used in this report. 


Objectives 


The primary audit objective was to evaluate the overall management of waivers of 
developmental and operational test requirements. Specifically, we evaluated the 
process for justifying, reviewing, and approving the waiver of developmental and 
operational testing requirements for Defense systems. Additionally, we evaluated 
the management control program as it related to the audit objective. See 
Appendix A for a discussion of the scope and methodology, the review of the 
management control program, and prior coverage related to the audit objectives. 


A. Meeting Operational Requirements 


The independent operational test and evaluation organizations within the 
Military Departments were not able to fully resolve critical operational 
issues (COIs) for weapon systems during initial operational test and 
evaluation. This condition occurred because the Military Departments 
were approving operational test requirement waivers or waiver 
equivalents, such as test limitations and deferrals, so that: 


e the full-rate production decisions would not be delayed and 


e program tests and evaluations would be deferred into the 
production phase of the acquisition process. 


As a result, weapon systems that were not fully ready continued into 
production before critical operational issues were resolved, creating a 
potential need for costly retrofit of fielded units and an initial operational 
performance that may be less than required to defeat the expected threat. 


Test and Evaluation Policy 


The following provides an overview of DoD and Military Department policy 
concerning COIs and operational testing for acquisition programs. A COI is an 
issue of operational effectiveness and operational suitability, normally phrased as 
a question, that must be examined in operational test and evaluation to determine 
the system’s capability to perform its mission. Appendix C provides a detailed 
discussion of the policy. 


DoD Policy. The DoD Instruction 5000.2, “Operation of the Defense Acquisition 
System,” May 12, 2003, requires that deficiencies encountered in testing before 
the low-rate initial production (LRIP) decision be resolved before the system can 
proceed beyond LRIP and that any fixes to those deficiencies be verified during 
follow-on operational test and evaluation.’ In addition, the DOT&E issued 
guidance on August 3, 2001, in a memorandum, “DOT&E Goals,” that directed 
the Military Department testers not to issue waivers or deferrals that would 
prevent a complete evaluation of operational requirements. The DOT&E 
guidance also states that test requirements that have been waived should be 
considered failed unless the ORD requirement is changed. 


The previous DoD 5000 series guidance stated that, before the full-rate 
production decision review, operational test and evaluation will: 


e determine the operational effectiveness and suitability of a system 
under realistic conditions, 


e determine whether the thresholds and objectives in the approved ORD 
and the COIs were satisfied, and 


'The previous draft DoD Instruction had the same requirement. The Office of the Secretary of Defense 
canceled the DoD 5000 series guidance and replaced it with 120-day interim guidance in October 2002. 


e assess the effects of demonstrated system performance on combat 
operations. 


The guidance also stated that the milestone decision authority will schedule a 
review during the production and deployment phase to consider the results of 
qualification testing and initial operational test and evaluation before authorizing 
full-rate production and deployment. 


Army Policy. The policy states that COIs are the operational effectiveness, 
suitability, and survivability concerns that must be examined in operational test 
and evaluation to determine the degree to which the system is capable of 
performing its mission. 


Navy Policy. The policy states that operational test and evaluation is subdivided 
into initial operational test and evaluation and follow-on operational test and 
evaluation. For each program, the Commander, Operational Test and Evaluation 
Force (COMOPTEVFOR): 


e develops the COIs to be addressed, 
e publishes the COIs in the TEMP, and 


e links the COIs to the Chief of Naval Operations requirements 
established in the ORD. 


Air Force Policy. The policy states that the use of operational test and evaluation 
will determine the operational effectiveness and suitability of systems undergoing 
research and development efforts. The initial operational test and evaluation 
substantiates decisions for proceeding beyond LRIP, full-rate production and 
fielding, and declaration of initial operational capability. The results of initial 
operational test and evaluation are to answer completely and unambiguously all 
COls as thoroughly as possible. Further, testers are not to defer testing to 
follow-on operational test and evaluation unless unavoidable. 


Addressing Critical Operational Issues 


The Military Departments’ acceptance of test waivers and other test limitations 
impeded their independent operational test and evaluation organizations from 
fully resolving COIs during initial operational test and evaluation. This condition 
occurred because the Military Departments were approving operational test 
requirement waivers or waiver equivalents, such as test limitations and deferrals, 
so that: 


e the full-rate production decisions would not be delayed and 


e program tests and evaluations would be deferred into the production 
phase of the acquisition process. 


DOT&E identified 11 DoD programs with approved operational test requirement 
waivers or waiver equivalents dated from October 1, 1997, through May 6, 2002. 
Our review of those 11 programs indicated that 8 of the respective program 
offices entered initial operational test and evaluation with deficiencies and 
postponed testing and evaluation that affected the resolution of one or more COIs 
for operational effectiveness or operational suitability. Appendix D describes the 
11 programs reviewed. 


The following table summarizes the system-specific COIs affected by test waivers 
or waiver equivalents for the eight systems. The COIs affected by test waivers 
and limitations are categorized by operational effectiveness and operational 
suitability. The table further separates operational suitability into the areas of 
reliability, availability, maintainability, interoperability, compatibility, logistics 
supportability, safety, human factors, and training. The table identifies the COI 
categories not resolved because of test waivers or waiver equivalents for the 

eight programs affected as indicated by the “X.” The nine COI categories listed 
are not applicable to each of the eight programs. 


Critical Operational Issues Not Fully Resolved Because of Approved Test Waivers 


Army 


Javelin 


or Test Limitations for the Eight Defense Systems Affected 


Critical Operational Issues Categories 
Operational 
Effectiveness Operational Suitability 
Logistics 
Reli- Avail- Maintain-  Inter- Compat- Support- Human 
ability ability ability operability ibility ability Safety Factors 


x 


Patriot Advanced 
Capability x x x 
Tactical Unmanned 
Aerial Vehicle Xx x 


Navy 


F/A-18E/F Super Hornet ” 
V-22 Osprey* 


Air Force 


F-15 Tactical Electronic 


Warfare System 
Fighter Data Link 


x» 


Predator Unmanned 
Aerial Vehicle x xX xX 


“Deliberative process privilege data omitted. 


°The Navy testers considered the COIs to be resolved even though the program office had not resolved the 
deficiencies that caused the test waivers before the full-rate production decision. 


A discussion of the nine COI categories affected by test waivers and limitations 
and examples of the affected COIs from the eight programs affected follow. 
Further details of all affected COIs are in Appendix E. 


Operational Effectiveness 


Of the eight programs affected by waivers, five programs deferred planned 
operational tests that affected the resolution of COIs associated with operational 
effectiveness. Examples of those programs are the Patriot Advance Capability-3, 
the F/A-18E/F Super Hornet, and the Predator Unmanned Aerial Vehicle. 


Patriot Advance Capability-3. The COI requires the Patriot Advanced 
Capability-3 to enhance the effectiveness of the Patriot system. The testers were 
not able to resolve the operational effectiveness COI before the planned full-rate 
production decision because the Army deferred test requirements: 


e for the missile to effectively destroy a target and counteract 
countermeasures, 


e for the ground system to distinguish between targets, and 
e for the system to operate in all required environments. 


In consideration of performance problems identified during limited user testing, 
the Program Manager obtained the approval of the Joint Requirements Oversight 
Council to defer those test requirements as part of an evolutionary development 
program after the planned full-rate production decision. As a consequence, the 
Army Test and Evaluation Command determined in the limited user tests 
preceding the LRIP decision that the Patriot Advanced Capability-3 had limited 
operational effectiveness. To subsequently demonstrate those deferred test 
requirements, the Program Office stated that the Army increased funding for 
system development by approximately $90 million to demonstrate those ORD 
requirements. During the initial operational test and evaluation conducted in 2002, 
the Army Test and Evaluation Command and the Director, Operational Test and 
Evaluation determined that the Patriot Advanced Capability-3 was operationally 
effective with limitations. 


F/A-18E/F Super Hornet. 


"Deliberative process privilege data omitted. 


* 


The issue of the F/A-18E/F waivers, 
limitations, and deficiencies that affected the resolution of operational 
effectiveness and suitability COIs before the planned full-rate production decision 
has also been addressed in reports by the Inspector General of the Department of 
Defense (IG DoD) and the Defense Science Board. 


Inspector General of the Department of Defense. The IG DoD Report 
No. 99-205, “Operational Testing and Evaluation of the F/A-18E/F Super Hornet,” 
July 7, 1999, states that, in March 1999, the COMOPTEVFOR submitted the test 
plan for the F/A-18E/F. The test plan listed one limitation that precluded it from 
assessing the COI for reconnaissance because the tactical reconnaissance sensor 
suite was not cleared for employment during the operational evaluation. In 
addition, the report states that, before the operational evaluation began in May 
1999, the Navy approved a temporary waiver for uncorrected technical 
deficiencies identified during developmental testing before beginning initial 
operational test and evaluation. 


Defense Science Board. Report of the Defense Science Board Task Force 
on Test and Evaluation Capabilities, December 2000, states that the Navy did not 
assess the effect of waived test requirements on the resolution of the system’s 
COIs during the evaluation of F/A-18E/F aircraft operational effectiveness and 
suitability. The Navy waived several of the test requirements because certain 
aircraft components exhibited poor reliability during developmental testing. 
Accordingly, the Navy did not assess the effect of those aircraft components and 
the associated repair times in the evaluation of the key F/A-18E/F aircraft 
suitability requirements of reliability, maintainability, and availability. Several of 
the waived aircraft components, which the Navy did not include in the operational 
assessment, caused failures and required a significant amount of time for 
inspections and repairs during the operational evaluation. 


Predator Unmanned Aerial Vehicle. Although the Predator System Program 
Office certified that the Predator was ready to enter operational testing, it 
identified operational effectiveness limitations related to the system’s ability to 
recognize tactical-sized targets. Those limitations were identified during informal 
developmental testing. An operational effectiveness COI for the Predator requires 
it to perform reconnaissance, surveillance, and target acquisition tasks to support 
battle management, execution, and operations other than war. Based on 
operational testing conducted before the full-rate production decision, DOT&E 
stated that the Predator was not operationally effective, which included the 
inability to recognize tactical-sized targets. 


Reliability, Availability, and Maintainability 


Because reliability, availability, and maintainability are interrelated, we combined 
them for discussion purposes. Of the eight programs affected by waivers, 

seven programs deferred planned operational tests that affected the resolution of 
COIs associated with suitability in the areas of system reliability, availability, 


“Deliberative process privilege data omitted. 


maintainability, or a combination thereof. Examples of those programs are the 
Javelin, the V-22 Osprey, and the F-15 Tactical Electronic Warfare System. 


Javelin. A COI for the Javelin requires that it be designed so that it is sufficiently 
reliable and maintainable to employ on the projected battlefield. However, the 
Javelin entered initial operational test and evaluation with limitations associated 
with reliability. Engineering and manufacturing development testers estimated 
that the field tactical trainer (range) would achieve 25 hours, rather than the 

100 hours required. The system completed initial operational test and evaluation 
in December 1993, the results of which indicated that the system had unresolved 
operational suitability issues related to the reliability parameters prescribed for 
system maturity; specifically, the training devices were unreliable. According to 
the Deputy Under Secretary of the Army (Operations Research), the U.S. Army 
Infantry reevaluated the requirements in 1994 and reduced the 100-hour 
requirement to 50 hours that should be attained at the time of system maturity, 

3 years after the full-rate production decision. The Deputy also stated that the 
requirement for the full-rate production decision was 32 hours as shown in the 
System Evaluation Report, March 27, 1997. The Army operational testers 
resolved the operational suitability issues after the full-rate production decision. 


V-22 Osprey. A COI for the V-22 requires that it be reliable and maintainable in 
its intended operating environment and available to support its mission. 
Developmental test results indicated that the V-22 mean time between failure and 
false alarm rate did not achieve technical thresholds, thereby affecting the 
complete resolution of reliability, availability, maintainability, and diagnostic 
issues with the V-22 during operational test and evaluation. Consequently, the 
program manager requested a waiver of the V-22 mean time between failure and 
false alarm rate requirement. The Program Executive Officer for Air 
Anti-Submarine Warfare, Assault, and Special Mission Programs stated that the 
waived requirement would be ready for operation testing after the full-rate 
production decision. On November 17, 2000, DOT&E stated in its beyond LRIP 
report for the V-22 that the V-22 was not operationally suitable, primarily because 
of reliability, maintainability, availability, human factors, and interoperability 
issues. DOT&E based its conclusion on the failure rate of aircraft components 
exhibited during testing. 


F-15 Tactical Electronic Warfare System. A COI for the F-15 Tactical 
Electronic Warfare System (the System) requires that the Band 1.5 system be 
operationally ready to support F-15E mission requirements. The Air Force will 
depend on the capability of the built-in-test of the System to identify faults and 
take corrective action in a timely manner. The May 2000 TEMP did not show any 
test limitations associated with the built-in-test; however, it did show that the 
System had an open deficiency for an excessive built-in-test false alarm rate. 
Although the deficiency existed, the Air Force conducted initial operational test 
and evaluation in FY 2000 and determined that the built-in-test false alarm rate 
was 65 percent, compared to the required built-in-test false alarm rate of less than 
20 percent. The excessive false alarm rate caused numerous unwarranted 
maintenance actions and distracted the air crew’s attention from primary mission 
functions. Even with the excessive false alarm rate, the Air Force approved the 


system for production because the System provided a greater capability than was 
previously available, according to DOT&E representatives. In addition, DOT&E 
approved the TEMP and operational test and evaluation plan for the Band 1.5 
system, both of which addressed the false alarm rate and required future testing, 
according to Air Force Test and Evaluation Directorate representatives. The 
representatives also stated that the Air Force submitted plans to DOT&E to correct 
the false alarm rate problems in the next increment, Band 5.0. 


Interoperability 


The Predator Unmanned Aerial Vehicle exempted planned operational tests of 
interoperability data links and file formats because the Joint Interoperability Test 
Command had not certified the system for interoperability. This exemption 
affected the resolution of the COI requiring the Predator to be interoperable with 
other command, control, communication, and computers and intelligence systems. 
Accordingly, the System Program Office certified that the Predator Unmanned 
Aerial Vehicle was ready for operational testing with a stipulation that, if the Joint 
Interoperability Test Command did not certify the system for interoperability 
during initial operational test and evaluation, the System Program Office would 
complete the certification process after the completion of operational testing. 
According to the DOT&E FY 2001 Annual Report, the Joint Interoperability Test 
Command had not yet certified the Predator for interoperability because of the 
untested requirements affecting the resolution of the COI. Further, the Joint 
Interoperability Test Command had certified only three of the seven critical system 
interfaces because the remaining interfaces were not available for evaluation. The 
System Program Office deferred further interoperability testing of the Predator 
until the Military Departments field ground stations that are equipped with tactical 
control stations. 


Compatibility 


Of the eight programs affected by waivers, the program offices for the F/A-18E/F 
Super Hornet, and the V-22 Osprey deferred planned operational tests that affected 
the resolution of COIs associated with suitability in the area of system 
compatibility. 


F/A-18E/F Super Hornet. On the F/A-18E/F Program, the Director of Navy Test 
and Evaluation and Technology Requirements, Office of the Chief of Naval 
Operations approved four test waiver requests that affected the resolution of the 
COI requiring the F/A-18E/F to be compatible with its operating environment. 

The Program Executive Officer for Tactical Aircraft requested the waivers because 
developmental test results evidenced poor communication between aircraft, 
inadequate radio reception while transmitting, ineffective concurrent transmission, 
and incompatibility with the combat training system. In the request, the Program 
Executive Officer for Tactical Aircraft stated that the waived requirements would 
be subjected to operational testing after the planned full-rate 


production decision. 


V-22 Osprey. On the V-22 Osprey Program, the Director of Navy Test and 
Evaluation and Technology Requirements approved three test waivers that the 
program manager requested for the COI requiring the V-22 to be compatible with 
its operating environment. The program manager requested the waivers because: 


e developmental test results identified inadequate cockpit and cabin 
nuclear, biological and chemical overpressure protection; an inability to 
align the lightweight inertial navigation system without the global 
positioning system signal; and 


e the aircraft could not be cleared to operate in icing conditions. 


The Program Executive Officer for Air Anti-Submarine Warfare, Assault, and 
Special Mission Programs stated that the waived requirement would be ready for 
operational testing after the full-rate production decision. 


Logistics Supportability 


The Program Office for the Tactical Unmanned Aerial Vehicle deferred planned 
operational tests that affected the resolution of the COI requiring the system to 
meet the ground maneuver brigade commander’s sustained combat requirements. 
Specifically, the test and evaluation was limited because testers could not verify 
the supportability of the system when using heavy fuel. The ORD objective 
requires the system to use only heavy fuel; however, the system available for 
testing used motor gasoline, which meets the operational requirement threshold, 
the minimal acceptable value necessary to satisfy the need. According to 
DOT&E, even though the system used motor gasoline, the system met the fuel 
requirement by meeting the threshold; however, the Army is addressing 
supportability issues about the system being able to support a motor gasoline in the 
future. After the planned full-rate production decision, the Army will conduct a 
follow-on operational test and evaluation to test for the heavy fuel requirement. 


Safety 


Of the eight programs affected by waivers, the Program Offices for the F/A-18E/F 
Super Hornet and the V-22 Osprey deferred planned operational tests that affected 
the resolution of COIs associated with suitability in the area of system safety. 


F/A-18E/F Super Hornet. On the F/A-18E/F Program, the Director of Navy Test 
and Evaluation and Technology Requirements approved three waiver requests, 
which the Program Manager submitted, that affected the resolution of a COI 


* Deliberative process privilege data omitted. 
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related to safety. The COI requires that the F/A-18E/F be safe to operate and 
maintain. Based on developmental testing results, the program manager requested 
that operational tests addressing the hydraulic fluid level indicator and erroneous 
velocity vector performance be waived. The Program Executive Officer for 
Tactical Aircraft stated that the waived requirement would be subject to 
operational testing after the full-rate production decision. 


* 


V-22 Osprey. On the V-22 Osprey Program, the Director of Navy Test and 
Evaluation and Technology Requirements approved four test waivers, which the 
Program Manager requested, that affected the resolution of the COI requiring the 
V-22 to be safe to operate and maintain. The program manager requested the 
waivers because, during developmental testing, excessive force was required to 
disconnect the intercommunication system, the pilot and copilot seats had 
nonqualified inertial reels, the autorotative descent could not be maintained while 
attempting engine air start, and the ground collision avoidance and warning system 
was not available for test. 


Human Factors 


For the eight programs affected by waivers, the Program Office for the F/A-18E/F 
Super Hornet deferred planned operational tests that affected the resolution of the 
COI requiring that the human factors aspects of the F/A-18E/F support completion 
of the aircraft’s mission. The Program Executive Officer for Tactical Aircraft 
requested and the Director of Navy Test and Evaluation and Technology 
Requirements approved a waiver of the test for the ALR-67 power switch on the 
center pedestal because it was not within reach of the pilot when he was strapped 
in the seat. The Program Executive Officer for Tactical Aircraft stated that the 
waived requirement would be subject to operational testing after the full-rate 
production decision. 


* Deliberative process privilege data omitted. 
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Effect of Not Meeting Operational Requirements 


Without the Military Departments fully resolving COIs for weapon systems 
during initial operational test and evaluation, weapon systems that were not fully 
ready continued into production, thereby creating a potential need for costly 
retrofit of fielded units and an initial operational performance that may be less 
than required to defeat the expected threat. 


Management Comments on the Finding and Audit Responses 


Summaries of management comments on the finding and audit responses are in 
Appendix F for the Director, Operational Test and Evaluation; Appendix G for 
the Army; Appendix H for the Navy; and Appendix I for the Air Force. 


Recommendations, Management Comments, and Audit 
Responses 


A. We recommend that the Under Secretary of Defense for Acquisition, 
Technology, and Logistics, in conjunction with the Director, Operational 
Test and Evaluation, require that: 


1. The Military Departments assess operational performance 
requirements, including critical operational issues, as unattained in test 
reports when test waivers, or their equivalents, prevent a complete 
evaluation of operational requirements. 


Under Secretary of Defense for Acquisition, Technology, and Logistics 
Comments. The Under Secretary did not provide comments on the draft report. 
We request that the Under Secretary provide comments in response to the final 
report. 


Director, Operational Test and Evaluation Comments. The Director did not 
comment on the recommendation. We request that the Director provide 
comments in response to the final report. 


Deputy Under Secretary of the Army (Operations Research) Comments. 
Although not required to comment, the Deputy agreed with the recommendation, 
stating that the Army focused its COIs and criteria on mission accomplishment. 
Further, he stated that a breach of a specific criterion is a reason to delay entry 
into full-rate production unless other evidence shows that the system is 
operationally effective and suitable. For the complete text of the Deputy’s 
comments, see the Management Comments section of the report. 
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Deputy Assistant Secretary of the Navy (Research, Development, Test and 
Evaluation) Comments. Although not required to comment, the Deputy 
disagreed with the recommendation, stating that the Military Departments and the 
milestone decision authorities already conduct assessments of the operational 
performance requirements and consider the assessments in their program 
decisions. Further, he stated that, if the intent of the recommendation is to 
prevent the Military Departments or the milestone decision authorities from 
fulfilling their obligations until the testers operationally test all items, then such a 
recommendation should be rescinded. The Deputy also stated that the 
Component Acquisition Executives and milestone decision authorities are fully 
aware of and consider test waivers and deferrals as well as test results in their 
decisions. The Deputy asserted that the report provides no cases where test 
waivers or deferrals resulted in a Component Acquisition Executive or a 
milestone decision authority making uninformed decisions that caused costly 
retrofits of capability due to inadequate tests or that resulted in a program’s not 
meeting the expected threat. The Deputy stated that his comments contained 
information considered privileged in litigation, primarily under the deliberative 
process privilege and, therefore, was exempt from mandatory disclosure under the 
Freedom of Information Act. Where applicable, he marked his comments “For 
Official Use Only.” For the complete text of the Deputy’s comments, see the 
Management Comments section of the report. 


Audit Response. The recommendation is not intended to prevent Military 
Departments or milestone decision authorities from fulfilling their obligations 
until all items are operational tested. The intent of the recommendation was to 
ensure that the milestone decision authorities are fully aware of any COIs that the 
respective program offices may have waived, deferred, or limited that could affect 
the operational effectiveness or operational suitability of the system and its ability 
to meet the users’ requirements. 


A milestone decision authority cannot be fully aware of and consider test waivers 
and deferrals as well as test results if an assessment of the waiver, deferral, or 
limitation is not available at the time of the milestone decision. In making a 
milestone decision, the milestone decision authority is relying on the respective 
program office to satisfactorily resolve the waived COIs based on future test 
results and on the resolution of those COIs that will not result in costly retrofits 
for the system to fully meet operational requirements and user needs. 


Resolution of waived COIs can result in costly retrofits in systems produced, 
however. The report did not illustrate cases where test waivers or deferrals 
resulted in the milestone decision authority making uninformed decisions that 
caused costly retrofits or that resulted in a program not meeting the expected 
threat. However, the Defense Science Board Report, December 2000, concluded 
that the process of handling waivers seriously undermined the test and evaluation 
process and may have negatively affected weapon systems. Further, the DOT&E 
stated in his annual report for FY 2002 that he was concerned about the pressure 
to deploy new systems that the Military Departments have not adequately tested. 


if) 


He stated that the desire to field new capabilities as soon as possible should be 
tempered with the responsibility to ensure that associated weapon systems will 
not put Americans at risk. He also noted that, because of concern about fielding 
of inadequately tested systems, Congress has enacted new legislation requiring 
the DOT&E annual report to identify Military Departments’ waivers or deviations 
from testing requirements. 


Director, Air Force Test and Evaluation Comments. Although not required to 
comment, the Director disagreed with the recommendation, stating that the terms 
“waivers or their equivalents” are too problematic and that the Office of the 
Secretary of Defense (OSD) and the Military Departments have not uniformly 
accepted the terms or practices associated with that terminology. Further, he 
stated that the DOT&E did not coordinate his August 3, 2001, memorandum with 
the Air Force, and that the Air Force has fundamental disagreements with the 
memorandum’s statements on waivers and requirements. For the complete text of 
the Director’s comments, see the Management Comments section of the report. 


Audit Response. Because the terms “waivers or their equivalents” vary among 
the Military Departments, we have recommended in Finding B that the OSD and 
the Military Departments collectively establish consistent guidance for test 
waivers or deferrals. In addition, the DOT&E stated in his August 3, 2001, 
memorandum that he: 


e intended to communicate the major points of the memorandum with 
the commanders of the operational test agencies and 


e encouraged the commanders to share the guidance with their staffs. 


Further, in January 2003, the DOT&E stated that the Military Departments should 
have already promulgated the test policy in his August 3, 2001, memorandum. 


2. Milestone decision authorities require that program managers 
resolve critical operational issues before approving full-rate production for a 
weapon system. 


Under Secretary of Defense for Acquisition, Technology, and Logistics 
Comments. The Under Secretary did not provide comments on the draft report. 
We request that the Under Secretary provide comments in response to the final 
report. 


Director, Operational Test and Evaluation Comments. The Director partially 
concurred, stating that examples may exist where, because of an operational 
requirement, a capability is urgently needed. Consequently, testers may not fully 
evaluate all COIs during operational test and evaluation because of unanticipated 
problems, such as test range limitations, instrumentation limitations, or 
availability. In such a case, he stated that the milestone decision authorities 
should have the prerogative to field the current capability, which would be more 
beneficial to the warfighter, rather than wait to enter full-rate production until the 
respective program office resolves all COIs. 
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Audit Response. The Director’s comments on the recommendation were 
responsive. We agree that in the excepted case where a capability is needed 
because of an urgent operational requirement, the milestone decision authorities 
should have the prerogative to field the current capability, rather than wait to 
enter full-rate production until the respective program office resolves all COIs. 
Otherwise, milestone decision authorities need to comply with the DoD 
Instruction 5000.2 requirement that deficiencies encountered in testing before the 
LRIP decision be resolved before the system can proceed beyond LRIP, and that 
any fixes to those deficiencies be verified during follow-on operational test and 
evaluation. 


Deputy Under Secretary of the Army (Operations Research) Comments. 
Although not required to comment, the Deputy agreed with the concept of the 
recommendation and provided the same comments as he did for 
Recommendation A.1. 


Deputy Assistant Secretary of the Navy (Research, Development, Test and 
Evaluation) Comments. Although not required to comment, the Deputy 
disagreed with the concept of the recommendation and provided the same 
comments as he did for Recommendation A.1. 


Audit Response. See the audit response for Recommendation A.1. In addition to 
the earlier response, DoD Instruction 5000.2, requires that deficiencies 
encountered in testing before the LRIP decision be resolved before the system can 
proceed beyond LRIP, and that any fixes to those deficiencies be verified during 
follow-on operational test and evaluation. 


Director, Air Force Test and Evaluation Comments. Although not required to 
comment, the Director disagreed with the recommendation, stating that the 
recommendation was counter-productive and undermines the Under Secretary of 
Defense for Acquisition, Technology, and Logistics’ new direction in the 

DoD 5000-series to make evolutionary acquisition the preferred acquisition 
strategy. The Director believed that the milestone decision authorities must retain 
sufficient management flexibility to plan wisely and must retain authority to make 
needed trade-offs among cost, schedule, and performance for the good of the 
program. Further, he believed that large cost overruns and long delays to fielding 
new capabilities to warfighters would become routine if DoD implemented the 
recommendation. 


Audit Response. The key point of this report is to not just field new capabilities, 
but to field new capabilities that will fully meet warfighter requirements by 
efficiently resolving core impediments to fulfilling those requirements. A system 
being acquired using the evolutionary acquisition strategy still has to meet the 
operational requirements and COIs for that block of the evolutionary acquisition. 
When milestone decision authorities make trade-offs among cost, schedule, and 
performance, they still need to ensure that the system will meet user requirements. 
Therefore, system deficiencies that were identified before the system entered 
initial operational test and evaluation should be resolved before the system enters 
full-rate production for that block. The resolution of pre-initial operational test 
and evaluation deficiencies before full-rate production does not conflict with the 
DoD Instruction 5000.2, which requires that deficiencies encountered in testing 
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before the LRIP decision be resolved before the system proceeds beyond LRIP. 
Enforcement of this requirement will avoid large cost overruns to retrofit fielded 
systems that do not meet user requirements. Further, timely management efforts 
to resolve system deficiencies before production should not significantly delay 
fielding new capabilities to warfighters. On the other hand, it will provide the 
warfighters with a system that is operationally effective and suitable to defeat the 
expected threat. 
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B. Waiver Policies and Procedures 


The Military Departments applied inconsistent waiver-and-limitation 
terminology and procedures when referring to the deferral of testing. 
Although the Military Departments have policies and procedures for 
waivers, only the Navy processed waivers to defer testing of operational 
requirements. Conversely, the Army and the Air Force cited test 
limitations to defer testing of operational requirements. This condition 
occurred because OSD had not issued specific guidance for approving and 
processing waivers and other deferrals of operational testing. 
Subsequently, the OSD issued guidance directing the Military Department 
testers not to issue waivers or deferrals that would prevent a complete 
evaluation of operational requirements and require deficiencies 
encountered in testing to be resolved before LRIP. However, the Military 
Departments have not incorporated the new guidance into their testing 
policies and procedures. Without consistent and updated 
waiver-and-limitation terminology and procedures for deferral of testing, 
the Military Departments’ use of test waivers is inconsistent with OSD 
policy and affects the milestone decision makers’ ability to make fully 
informed decisions before systems enter LRIP and full-rate production. 


Military Departments’ Application of Waiver Policy and 
Procedures 


The Military Departments inconsistently applied waiver-and-limitation 
terminology and procedures when referring to deferral of testing until after the 
full-rate production decision review. Specifically: 


e the Army procedures require waivers when testing in the approved 
TEMP will not be conducted and use the terms “limitations” and 
“delays” to defer specific operational test requirements, 


e the Navy procedures require waivers to defer specific operational test 
requirements, and 


e the Air Force procedures require waivers to defer specific operational 
test requirements; however, instead of using waivers, program 
managers used test limitations to defer specific operational test 
requirements. 


Army Procedures. Army Regulation 73-1, “Test and Evaluation Policy,” 
January 7, 2002, and Army Pamphlet 73-2, “Test and Evaluation Master Plan 
Procedures and Guidelines,” October 11, 1996, address developmental and 
operational testing waivers and other methods for deferring testing. 


Testing Waivers. Army Regulation 73-1 requires testers to conduct the 


testing specified in the approved TEMP unless the program manager or other 
appropriate official submits a written request for waiver and receives approval 
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from the TEMP approval authority. Further, the Regulation states that the 
approval of testing waivers will depend on the availability and acceptability of 
relevant data and information and will not negate the requirement for independent 
system evaluation. For those tests for which waivers were granted for reasons 
other than availability of data, the program manager is to expand production 
testing or follow-on operational testing to address those operational requirements 
not previously tested. 


Testing Deferrals. Army guidance uses the terms “limitations” and 
“delays” to defer specific operational test requirements. Further, Army guidance 
requires the approved TEMP to state the test limitations that may affect the 
resolution of critical operational issues; the impact of those limitations, such as 
the effects on COIs; and the ability to formulate conclusions regarding 
operational effectiveness and operational suitability. In addition, Army 
Regulation 73-1 states that developmental or operational testing may be delayed 
when circumstances warrant. Test delays can be issued at the start of testing to 
address evaluation issues when the tester identifies a problem that would affect 
the completeness of the data being collected. Further, the Regulation states that 
tests will be delayed when it is apparent that the system has little chance of 
successfully attaining critical technical parameters or satisfying critical 
operational criteria, and when deficiencies cannot be resolved before the start of 
test. 


Navy Procedures. Secretary of the Navy Instruction 5000.2B, “Implementation 
of Mandatory Procedures for Major and Non-Major Defense Acquisition 
Programs and Major and Non-Major Information Technology Acquisition 
Programs,” December 6, 1996, addresses processing Navy waivers that affect the 
dedicated initial operational test and evaluation. Navy waivers consist of waivers 
from compliance with the criteria for certification of readiness for operational test 
and evaluation and waivers for deviations from TEMP testing requirements. The 
Navy processes those waivers in the following manner, with the waived 
requirement being tested in subsequent operational tests. 


Waiver Process. The program manager or program executive officer 
formulates waiver requests before the operational test readiness and certification 
review process in an operational test and evaluation certification message. When 
requesting a waiver, the program manager will outline the limitations that the 
deferral or waiver will place upon the system under test and the potential effects 
on fleet use. Further, a statement will be made in the operational test and 
evaluation certification message noting when the approved deferrals will be 
available for subsequent operational testing. The Director, Navy Test and 
Evaluation and Technology Requirements approves waiver requests, as 
appropriate, in coordination with the COMOPTEVFOR; the Deputy Chief of Naval 
Operations (Logistics); the Deputy Chief of Naval Operations (Resources, Warfare 
Requirements, and Assessments); and the program sponsor; and advises DOT&E 
when the waiver is for a major defense acquisition program. 


Subsequent Operational Tests. Approval of a test waiver request does 
not alter the test requirement. The program manager is still required to test the 
waived requirement in subsequent operational testing. A waiver may result in 
limitations to the scope of testing that preclude the COMOPTEVFOR from fully 
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resolving all COIs. The COMOPTEVFOR is not to assess waived requirements 
in its analysis to resolve COIs, but may comment on the waived requirements in 
the applicable test report. 


Air Force Procedures. Air Force Instruction 99-102, “Operational Test and 
Evaluation,” July 1, 1998; Air Force Operational Test and Evaluation Center 
Instruction 99-102, “Operational Test and Evaluation,” January 20, 2000; and 
Air Force Manual 63-119, “Certification of System Readiness for Dedicated 
Operational Test and Evaluation,” February 22, 1995, establish guidance for 
Air Force waivers from testing requirements. The Air Force procedures require 
program managers to process waivers to defer specific operational test 
requirements and identify test limitations in test plans. 


Test and Evaluation Waivers. Air Force Instruction 99-102 and 
Air Force Manual 63-119 require the Air Force Operational Test and Evaluation 
Center to prepare a request to waive or modify fundamental provisions for 
operational test and evaluation. With the user and developer concurrence, the 
Air Force Operational Test and Evaluation Center is to submit the request to the 
Air Force Test and Evaluation Directorate. The system program manager and the 
system’s program element monitor are to document any approved operational test 
and evaluation waivers in the TEMP and the program management directive, 
respectively. Air Force Manual 63-119 requires a waived requirement to be 
tested in subsequent operational test and evaluations or the user must change the 
ORD requirement. Before initial operational test and evaluation, the system 
program manager is to: 


e list any required waivers or areas excluded from the test and 
evaluation, the rationale, and future plans to clear the waivers, 
and 


e provide a summary of the list to the operational tester. 


In response, the operational tester is to indicate whether to proceed with the test 
and is to discuss the effects of test limitations and test waivers on resolving 
operational test issues in the test report. 


Other Deferred Testing. Air Force Instruction 99-102 states that the 
final operational test and evaluation report must contain descriptions of test 
methods and limitations; definitive test results; and system capabilities and 
limitations as measured against the ORD. According to Air Force Operational 
Test and Evaluation Center, testers should not include system limitations in the 
system evaluation and should not fail the system because of the limitations. 


OSD Guidance for Test Deferrals. The Military Departments did not 
consistently apply waiver-and-limitation terminology and procedures to defer 
required operational testing as discussed because the Under Secretary of Defense 
for Acquisition, Technology, and Logistics and the DOT&E had not issued 
guidance that provided specific procedures for requesting approval to defer test 
requirements to future periods and for evaluating those deferral requests. 
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Further, the DOT&E issued guidance on August 3, 2001, that directed Military 
Department testers not to issue waivers or deferrals that would prevent a complete 
evaluation of operational requirements. The DOT&E guidance also states that 
test requirements that have been waived should be considered failed unless the 
ORD requirement is changed. However, the Military Departments had not 
incorporated the new guidance into their testing policies and procedures. 


On May 12, 2003, OSD issued DoD Instruction 5000.2 that requires the 
operational test and evaluation testers to determine the operational effectiveness 
and suitability of a system under realistic operational conditions, including 
combat; determine whether thresholds in the approved ORDs’ and critical 
operational issues have been satisfied; and assess effects on combat operations. 

In addition, the Instruction requires that deficiencies encountered in testing before 
the LRIP decision be resolved before the system can proceed beyond LRIP, and 
that any fixes to those deficiencies be verified during follow-on operational test 
and evaluation. 


Consistently Applying Waiver-and-Limitation Terminology 
and Procedures 


Without consistent and updated waiver-and-limitation terminology and 
procedures for deferral of testing, the Military Departments’ use of waivers is 
inconsistent with OSD policy issued by the Under Secretary of Defense for 
Acquisition, Technology, and Logistics and the Director, Operational Test and 
Evaluation. Furthermore, the inconsistent policy affect the OSD milestone 
decision makers’ ability to make fully informed decisions before systems enter 
LRIP and full-rate production. 


Management Comments on the Finding and Audit Responses 


Summaries of management comments on the finding and audit responses are in 
Appendix F for the Director, Operational Test and Evaluation; Appendix G for 
the Army; Appendix H for the Navy; and Appendix I for the Air Force. 


DoD Instruction 5000.2 states that, during system development and demonstration, the capabilities 
demonstration document instead of the ORD will have the detailed operational performance parameters. 
Further, the Instruction states that the capabilities production document instead of the ORD will have the 
operational requirements resulting from system development and demonstration and will detail the 
performance expected of the production system. However, this report uses the term ORD because the 
programs reviewed during the audit used ORDs. 
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Recommendations, Management Comments, and Audit 
Responses 


Redirected and Revised Recommendation. In response to the recommendation, 
we received varied management comments; however, the general theme was the 
need for a coordinated corrective action plan to include input from the Under 
Secretary of Defense for Acquisition, Technology, and Logistics, and the 
DOT&E. Therefore, to better respond to the recommendation and to obtain 
succinct and attainable corrective action, we are redirecting the recommendation 
to include the Under Secretary of Defense for Acquisition, Technology, and 
Logistics, the DOT&E, and the respective Military Department organizations 
responsible for test and evaluation guidance. Further, in response to the 
comments by the Deputy Under Secretary of the Army (Operations Research), we 
revised the recommendation so that guidance for test waivers or deferrals would 
be jointly promulgated and implemented among OSD and the Military 
Departments. 


B. We recommend that the Under Secretary of Defense for Acquisition, 
Technology, and Logistics; the Director, Operational Test and Evaluation; 
the Deputy Under Secretary of the Army (Operations Research); the 
Assistant Secretary of the Navy (Research, Development, and Acquisition); 
and the Director, Air Force Test and Evaluation collectively establish 
consistent guidance for test waivers or deferrals. 


Deputy Under Secretary of the Army (Operations Research) Comments. The 
Deputy nonconcurred, stating that the DOT&E directed the guidance in his 
August 3, 2001, memorandum to his staff and focused on the completeness of 
evaluations. Further, he stated that applying consistent test waiver and deferral 
guidance throughout DoD requires the definition of terms. The Deputy suggested 
that we modify the recommendation to recommend that DoD establish a 
DoD-wide process action team, chaired by the Under Secretary of Defense for 
Acquisition, Technology, and Logistics and the DOT&E, with membership from 
the Military Department test and evaluation headquarters and the operational test 
agencies. By establishing the process action team, he concluded that the DoD 
would be able to establish consistent test waiver and test deferral guidance 
throughout DoD. For the complete text of the Deputy’s comments, see the 
Management Comments section of the report. 


Audit Response. The Deputy’s comments were responsive to the intent of the 
recommendation. Consequently, we redirected the recommendation to include 
the Under Secretary of Defense for Acquisition, Technology, and Logistics, the 
DOT&E, and the respective Military Department organizations responsible for 
test and evaluation guidance. The DOT&E stated that the guidance in his 
August 3, 2001, memorandum was communicated to the Military Departments, 
and that the Military Departments should already have promulgated the test 
policy to Military Department operational test organizations. 
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Deputy Assistant Secretary of the Navy (Research, Development, Test and 
Evaluation) Comments. The Deputy nonconcurred, stating that the Navy has 
formally stated and documented its policy on test waivers and deferrals and that 
the concerns expressed in the report are unfounded. He also stated that the Navy 
policy on test waivers and deferrals allows the Component Acquisition Executive 
and the milestone decision authorities the management flexibility needed to make 
programmatic decisions in a timely manner. The Deputy further stated that less 
management flexibility would add cost to programs without any benefit. The 
Deputy stated that his comments contained information considered privileged in 
litigation, primarily under the deliberative process privilege and, therefore, were 
exempt from mandatory disclosure under the Freedom of Information Act. Where 
applicable, he marked his comments “For Official Use Only.” For the complete 
text of the Deputy’s comments, see the Management Comments section of the 
report. 


Audit Response. Although the Navy has formally stated and documented its 
policy on test waivers and deferrals in Secretary of the Navy Instruction 5000.2B, 
the Senate Armed Service Committee in its conference report for the FY 2003 
DoD appropriation cited a need for the Navy to modify its guidance on waivers. 
Specifically, the Committee stated that the Defense Science Board, in its 
December 2000 Report, strongly recommended that the Navy modify Secretary of 
the Navy Instruction 5000.2B to prohibit it from issuing waivers without DOT&E 
review and approval. Further, as discussed in the associated conference report, 
Congress enacted new legislation requiring the DOT&E annual report to identify 
Military Departments’ waivers or deviations from testing requirements and the 
actions that the Military Departments have taken or plan to take to address the 
concerns that the waivers or deviations raised. 


The Navy policy on test waivers and deferrals allowing management flexibility 
states that, after the Director, Navy Test and Evaluation and Technology 
Requirements waives a COI or a portion thereof, the COMOPTEVFOR does not 
assess the associated requirements in its analysis of test results to determine 
whether the system has met the requirements of the COI. Consequently, based on 
the successful accomplishment of the tested portion of the unwaived COI 
requirements, the COMOPTEVFOR could assess and did assess the entire COI as 
met, even though that assessment did not include the waived requirements. 
Therefore, when the milestone decision authorities use the COMOPTEVFOR test 
analysis in their decision making process, they are informed that the system has 
met all of its COI requirements, to include the waived requirements, and that it is 
operationally effective and suitable and capable of performing its mission. 


The Deputy believes that less management flexibility would add cost to a 
program without benefit; however, greater flexibility could result in the Navy 
expending significant dollars to retrofit fielded units to meet its operational 
requirements if the milestone decision authority allows a system to enter 
production when the system has not met all of it COI requirements. 


Director, Air Force Test and Evaluation Comments. The Director stated that 
uniform policy for waivers may be good for DoD, but the Air Force nonconcurred 
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with making all Military Department policies, particularly Air Force policies, 
reflect the policies of the DOT&E. Further, he stated that DOT&E should first 
contact the Military Departments to seek agreement before establishing new 
policies. He believed that imposing a “one-size-fits-all” approach could 
undermine attempts by the Under Secretary of Defense for Acquisition, 
Technology, and Logistics to streamline acquisition using the evolutionary 
acquisitions and time-phased requirements in the new DoD 5000-series. 
However, the Director stated that the Air Force Test and Evaluation Directorate is 
rewriting and updating Air Force Instruction 99-101, “Developmental Test and 
Evaluation,” and Air Force Instruction 99-102, “Operational Test and 
Evaluation.” For the complete text of the Director’s comments, see the 
Management Comments section of the report. 


Audit Response. Without consistent and updated waiver-and-limitation 
terminology and procedures for deferral of testing, the Military Departments’ use 
of test waivers will continue to be inconsistent with OSD policy and could affect 
the milestone decision makers’ ability to make fully informed decisions before 
systems enter LRIP and full-rate production. The establishment of consistent 
terminology and procedures by the Military Departments should not affect using 
an evolutionary acquisition strategy. Conversely, the consistent terminology and 
procedures should strengthen the use of an evolutionary acquisition strategy by 
ensuring that systems meet the operational requirements and COIs for each block 
of the evolutionary acquisitions. 


We redirected and revised the recommendation so that the OSD and the Military 
Departments would jointly develop and implement guidance for test waivers and 
deferrals. 


Director, Operational Test and Evaluation Comments. Although not required 
to comment, the Director agreed with the concept of the recommendation and 
stated that DoD has already made progress in updating policies, practices, and 
procedures for managing waivers and that our administrative recommendations 
will help ensure proper management in that regard. For the complete text of the 
Director’s comments, see the Management Comments section of the report. 
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Appendix A. Scope and Methodology 


We reviewed documentation dated from June 1986 to January 2003. To 
accomplish the audit objective, we: 


discussed test and evaluation policies and procedures with personnel 
in the Office of the DOT&E, selected Military Department program 
offices, and operational test and evaluation organizations; 


assessed whether DOT&E and the Military Departments analyzed test 
and evaluation waiver in accordance with DoD and Military 
Department policy concerning critical operational issues and 
operational testing for acquisition programs; 


reviewed test reports that Military Department operational test and 
evaluation organizations issued for selected acquisition programs; 


reviewed approved requests for test and evaluation waivers to 
determine the effect on critical operational issues; 


reviewed TEMPs, including associated appendixes and attachments, 
Military Department developmental test reports, program budgets, and 
detailed program schedules for selected acquisition programs; 


identified the roles of the Office of the DOT&E and appropriate 
Military Department oversight groups in the DoD test and evaluation 
waiver process; and 


determined the role of the Institute for Defense Analyses in the test 
and evaluation waiver process. 


To perform the audit, we contacted the Office of the DOT&E and the testing 
organizations of the Military Departments to obtain a list of all programs for 
which they maintained oversight that had waivers issued from 1998 through May 
2002. The Office of the DOT&E provided a list of 11 programs that were issued 
waivers during the designated period. The programs were in various phases of 
the acquisition process and are described in Appendix D. The 11 programs 
reviewed were: 


the Army Javelin, the Patriot Advanced Capability-3, and the Tactical 
Unmanned Aerial Vehicle; 


the Navy Cooperative Engagement Capability, the F/A-18E/F Super 
Hornet, the Joint Stand-Off Weapon, and the V-22 Osprey; and 


the Air Force F-15 Tactical Electronic Warfare System, the Fighter 


Data Link; the Joint Direct Attack Munition, and the Predator 
Unmanned Aerial Vehicle. 
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We reviewed the 11 programs that the Office of DOT&E identified and obtained 
information for the F/A-18E/F Super Hornet and the V-22 Osprey from prior 
General Accounting Office and Inspector General of the Department of Defense 
reports. The Navy was the only Military Department that reported having 
processed waivers; however, it was unable to provide a list of those waivers. The 
Army and Air Force indicated that the programs for which they provided 
oversight had not issued waivers for the period reviewed. Our review of those 
11 programs indicated that 8 of the respective program offices entered initial 
operational test and evaluation with deficiencies and postponed testing and 
evaluation that affected the resolution of one or more COIs for operational 
effectiveness or operational suitability. The following are the eight programs: 


e the Army Javelin, the Patriot Advanced Capability-3, and the Tactical 
Unmanned Aerial Vehicle; 


e the Navy F/A-18E/F Super Hornet and the V-22 Osprey; and 


e the Air Force F-15 Tactical Electronic Warfare System, the Fighter 
Data Link, and the Predator Unmanned Aerial Vehicle. 


We performed this audit from June 2001 through February 2003 in accordance 
with generally accepted government auditing standards. 


Use of Computer-Processed Data. We did not rely on computer-processed data 
to perform this audit. 


General Accounting Office High-Risk Area. The General Accounting Office 
has identified several high-risk areas in the DoD. This report provides coverage 
of the DoD Weapon Systems Acquisition high-risk area. 


Management Control Program Review 


DoD Directive 5010.38, “Management Control (MC) Program,” August 26, 1996, 
and DoD Instruction 5010.40, “Management Control (MC) Program Procedures,” 
August 28, 1996, require DoD organizations to implement a comprehensive 
system of management controls that provides reasonable assurance that programs 
are operating as intended and to evaluate the adequacy of the controls. 


Scope of the Review of the Management Control Program. In accordance 
with DoD Directive 5000.1, “The Defense Acquisition System,” May 12, 2003, 
acquisition managers are to use program cost, schedule, and performance 
parameters as control objectives to implement the requirements of DoD 
Directive 5010.38. Accordingly, we limited our review to management controls 
directly related to the waivers of test and evaluation requirements. We did not 
assess management’s self-evaluation of those controls. 


Adequacy of Management Controls. The management controls were in place 


for review and approval of waivers; however, the Military Departments and the 
milestone decision authorities were not using the controls to ensure that DoD 
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testers addressed COIs that had waived tests, or other limitations, before the 
milestone decision authorities approved a weapon system for full-rate production. 
Implementation of Recommendations A.1. and A.2. will ensure that DoD testers 
address COIs that have waived tests, or other limitations, before a weapon system 
is approved for full-rate production. 


Prior Coverage 


During the last 5 years, the General Accounting Office and the Inspector General 
of the Department of Defense have issued reports that reference DoD test waivers 
and limitations. Unrestricted General Accounting Office and Inspector General of 
the Department of Defense reports can be accessed at http://www.gao.gov and 
http://www.dodig.osd.mil/audit/reports, respectively. 


General Accounting Office (GAO) 


GAO Report No. NSIAD-98-13, “Navy Aviation: V-22 Cost and 
Capability to Meet Requirements are yet to be Determined,” October 1997 


GAO Report No. NSIAD-00-199, “Best Practices: A More Constructive 
Test Approach is Key to Better Weapon System Outcomes,” July 2000 


GAO Report No. NSIAD-98-61, “Navy Aviation: F/A-18E/F 
Development and Production Issues, ” March 1998 


Inspector General of the Department of Defense (IG DoD) 


IG DoD Report No. D-2000-174, “V-22 Osprey Joint Advanced Vertical 
Aircraft,” August 15, 2000 


IG DoD Report No. 99-205, “Operational Testing and Evaluation of the 
F/A-18E/F Super Hornet,” July 7, 1999 
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Appendix B. Glossary 


Acquisition Category. An acquisition category is an attribute of an acquisition 
program that determines the program’s level of review, decision authority, and 
applicable procedures. The acquisition categories consist of I, major Defense 
acquisition programs; IA, major automated information systems; I, major 
systems; and III, all other acquisition programs. Acquisition Category I programs 
have two subcategories: ID and IC. Acquisition IA programs also have two 
subcategories: IAM and IAC. 


Availability. Availability is a measure of the degree to which an item is in an 
operable state and can be committed at the start of a mission when the mission is 
called for at an unknown (random) point in time. 


Compatibility. Compatibility is the capability of two or more items or 
components of equipment or material to exist or function in the same system or 
environment without mutual interference. 


Critical Operational Issue. A critical operational issue is an issue of operational 
effectiveness or operational suitability (not parameters, objectives, or thresholds), 
or both, that must be examined in operational test and evaluation to determine the 
system’s capability to perform its mission. A critical operational issue is 
normally phrased as a question that must be answered to properly evaluate 
operational effectiveness or operational suitability. 


Follow-On Operational Test and Evaluation. Follow-on operational test and 
evaluation is test and evaluation that is necessary during and after the production 
period to refine the estimates made during operational test and evaluation, to 
evaluate changes, and to reevaluate the system to ensure that it continues to meet 
operational needs and retains its effectiveness in a new environment or against a 
new threat. 


Human Factor. A human factor is the systematic application of relevant 
information about human abilities, characteristics, behavior, motivation, and 
performance. It includes principles and applications in the areas of human 
engineering, personnel selection, training, life support, job performance aids, and 
human performance evaluation. 


Initial Operational Capability. Initial operational capability is the first 
attainment of the capability to effectively employ a weapon, item of equipment, 
or system of approved specific characteristics with the appropriate number, type, 
and mix of trained and equipped personnel necessary to operate, maintain, and 
support the system. Initial operational capability is normally defined in the 
operational requirements document. 


Initial Operational Test and Evaluation. Initial operational test and evaluation 
is an operational test and evaluation conducted on production, or production 
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representative articles, to determine whether systems are operationally effective and 
suitable, and which supports the decision to proceed beyond LRIP. 


Interoperability. Interoperability is the ability of systems, units, or forces to 
provide services to or accept services from other systems, units, or forces and to 
use the services so exchanged to operate effectively together. 


Key Performance Parameters. Key performance parameters are capabilities or 
characteristics so significant that failure to meet the threshold or minimum 
acceptable value can be cause for the concept or system selected to be reevaluated 
or the program to be reassessed or terminated. 


Limited User Test. A limited user test is any type of research, development, test 
and evaluation funded user test conducted that does not address all of the 
effectiveness, suitability, and survivability issues and is therefore limited in 
comparison to an initial operational test that must address all effectiveness, 
suitability, and survivability issues. 


Logistics Supportability. Logistics supportability is the degree of ease to which 
system design characteristics and planned logistics resources (including the 
logistics support elements) allow for meeting system availability and wartime 
usage requirements. 


Low-Rate Initial Production. Low-rate initial production (LRIP) establishes an 
initial production base for the system, permits an orderly production-rate increase 
sufficient to lead to a smooth transition to full-rate production, and provides 
production representative articles for initial operational test and evaluation and 
full-up live fire testing. This work effort concludes with a full-rate production 
decision review to authorize full-rate production and deployment. 


Maintainability. Maintainability is the ability of an item to be retained in, or 
restored to, a specified condition when maintenance is performed by personnel 
having specified skill levels, using prescribed procedures and resources, at each 
prescribed level of maintenance and repair. 


Major System. A major system is a combination of elements that function 
together to produce the capabilities required to fulfill a mission need, including 
hardware, equipment, software, or any combination thereof, but excluding 
construction or other improvements to real property. A system will be considered 
a major system if it is estimated by the DoD Component Head to require an 
eventual total expenditure for research, development, test, and evaluation of more 
than $140 million in FY 2000 constant dollars, or for procurement of more than 
$660 million in FY 2000 constant dollars, or is designated as major by the DoD 
Component Head. Major systems are synonymous with Acquisition Category II 
programs. 


Operational Effectiveness. Operational effectiveness is the overall degree of 
mission accomplishment of a system when used by representative personnel in the 
environment planned or expected, such as natural, electronic, or threat, for 
operational employment of the system considering organization; doctrine; tactics; 
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survivability; vulnerability; and threat, including countermeasures, initial nuclear 
weapons effects, nuclear, biological, and chemical contamination threats. 


Operational Requirements Document. An operational requirements document 
(ORD) is a formatted statement, prepared by the user or user’s representative, 
containing performance and related operational performance parameters for the 
proposed concept or system. 


Operational Suitability. Operational suitability is the degree to which a system 
can be placed satisfactorily in field use with consideration being given to 
availability, compatibility, transportability, interoperability, reliability, wartime 
usage rates, maintainability, safety, human factors, manpower supportability, 
logistic supportability, natural environmental effects and impacts, documentation, 
and training requirements. 


Program. A program is an acquisition effort funded by research, development, 
test and evaluation or procurement appropriations, or both, with the express 
objective of providing a new or improved capability in response to a stated 
mission need or deficiency. 


Program Management Directive. The program management directive is an 

Air Force document, required for all Air Force acquisition programs, that directs 
acquisition responsibilities to the appropriate Air Force major commands, 
agencies, program executive offices, or designated acquisition commander. 


Reliability. Reliability is the ability of a system and its parts to perform its 
mission without failure, degradation, or demand on the support system. 


Safety. Safety is the freedom from conditions that can cause death, injury, 
occupational illness, damage/loss of equipment or property, or damage to the 
environment. 


Test and Evaluation Master Plan. The test and evaluation master plan (TEMP) 
documents the overall structure and objectives of the test and evaluation program. 
It provides a framework within which to generate detailed test and evaluation 
plans and it documents schedule and resource implications associated with the 
test and evaluation program. The TEMP identifies the necessary developmental 
test and evaluation, operational test and evaluation, and live fire test and 
evaluation activities. Further, the TEMP relates program schedule, test 
management strategy and structure, and required resources to critical operational 
issues; critical technical parameters; objectives and thresholds documented in the 
ORD; evaluation criteria; and milestone decision points. 


Third-Party Targeting. According to DOT&E representatives, third-party 
targeting allows a DoD asset other than the asset dropping the bomb (a third 
party) to control the coordinates for the target. Third-party targeting allows 
targeting while the aircraft is in the middle of its operations, unlike preplanned 
targeting, which requires setting the targeting coordinates before the mission. 
Third-party targeting provides for shorter update and quicker ability to change the 
target. 
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Waiver. According to DOT&E, a waiver is a deferral of an operational testing 
requirement and does not eliminate the requirement for subsequent testing. 
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Appendix C. Test and Evaluation Policy 


Deputy Secretary of Defense Memorandum, “Defense Acquisition,” October 30, 
2002; Army Regulation 73-1, “Test and Evaluation Policy,” January 7, 2002; 
Army Pamphlet 73-2, “Test and Evaluation Master Plan Procedures and 
Guidelines,” October 11, 1996, Secretary of the Navy Instruction 5000.2B, 
“Implementation of Mandatory Procedures for Major and Non-Major Defense 
Acquisition Programs and Major and Non-Major Information Technology 
Acquisition Programs,” December 6, 1996; Air Force Instruction 99-102, 
“Operational Test and Evaluation,” July 1, 1998; Air Force Operational Test and 
Evaluation Center Instruction 99-102, “Operational Test and Evaluation,” 
January 20, 2000; and Air Force Manual 63-119, “Certification of System 
Readiness for Dedicated Operational Test and Evaluation,” February 22, 1995, 
provide policy concerning operational testing and critical operational issues for 
acquisition programs. 


DoD Policy. On October 30, 2002, the Deputy Secretary of Defense issued a 
memorandum, “Defense Acquisition,” that canceled DoD Regulation 5000.2-R, 
“Mandatory Procedures for Major Defense Acquisition Programs (MDAPs) and 
Major Automated Information System (MAIS) Acquisition Programs,” 

April 5, 2002, and replaced it with interim guidance in the form of attachments to 
the memorandum. The interim guidance requires Military Departments to 
complete operational testing and evaluation before the system can proceed 
beyond the LRIP decision. Operational testing and evaluation is to determine the 
operational effectiveness and suitability of a system; determine whether the 
thresholds and objectives in the approved operational requirements document and 
the critical operational issues have been satisfied; and assess the effects of 
demonstrated system performance on combat operations. In addition, the 
independent operational test agencies are required to use production or 
production-representative articles for the dedicated phase of operational test and 
evaluation that support the full-rate production decision. The interim guidance 
also states that all developmental test and evaluation will identify technical 
capabilities and limitations. In addition to identifying test limitations, the 
developing agency must formally certify the system as ready for the dedicated 
phase of operational test and evaluation. 


Further, the interim guidance states that major Defense acquisition programs may 
not proceed beyond LRIP without approval by the milestone decision authority. 
The decision to continue beyond LRIP to full-rate production requires completion 
of initial operational test and evaluation and the submission of the beyond LRIP 
and live-fire test and evaluation reports, as applicable, to Congress, the Secretary 
of Defense, and the Under Secretary of Defense for Acquisition, Technology, and 
Logistics. 


The DOT&E issued guidance on August 3, 2001, that directed Military 
Department testers not to issue waivers or deferrals that would prevent a complete 
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evaluation of operational requirements. The DOT&E guidance also states that 
test requirements that have been waived should be considered failed unless the 
ORD requirement is changed. 


Army Policy. Army Regulation 73-1 defines critical operational issues and 
criteria as the operational effectiveness, suitability, and survivability concerns that 
must be examined in operational test and evaluation to determine the degree to 
which the system is capable of performing its mission. COIs are the key decision 
maker operational concerns or issues with standards of performance that must be 
answered by the system evaluation to determine if the system is ready to enter 
full-rate production. Further, COIs are associated with scope, criteria, and 
rationale. A breach of a criterion is reason for the milestone decision authority to 
delay entry of the system into full-rate production unless other evidence of 
acceptable system operational effectiveness and suitability is provided. Further, 
the Regulation states that testing specified in the approved TEMP must be 
conducted unless the program manager or other appropriate official submits a 
written request for waiver and the TEMP approval authority approves the waiver 
request. 


The Regulation also states that the approval of testing waivers will depend on the 
availability and acceptability of relevant data and information and will not negate 
the requirement for independent system evaluation. For those tests for which 
waivers were granted for reasons other than availability of data, the program 
manager is to expand production testing or follow-on operational testing to 
address those operational requirements not previously tested. 


The Regulation further states that testing may be delayed when circumstances 
warrant. Test delays can be issued at the start of testing to address evaluation 
issues when the tester identifies a problem that would affect the completeness of 
the data being collected. Further, the Regulation states that tests will be delayed 
when it is apparent that the system has little chance of successfully attaining 
critical technical parameters, satisfying critical operational criteria, or resolving 
deficiencies before the start of test. 


Army Pamphlet 73-2 requires the approved TEMP to state the test limitations that 
may affect the resolution of critical operational issues; the impact of those 
limitations, such as the effects on COIs; and the ability to formulate conclusions 
regarding operational effectiveness and operational suitability. 


Navy Policy. Secretary of the Navy Instruction 5000.2B states that operational 
test and evaluation is subdivided into initial operational test and evaluation and 
follow-on operational test and evaluation. Initial operational test and evaluation 
is all operational test and evaluation up to and including the completion of 
operational evaluation. For each program, the COMOPTEVFOR is to develop 
critical operational issues and publish them in the TEMP. The critical operational 
issues are to be linked to the Chief of Naval Operations requirements established 
in the operational requirements document. For programs requiring joint 
interoperability, joint interoperability critical operational issues must be 
established to address effectiveness during operational testing. Further, the 
Instruction provides guidance for processing waivers to defer testing of the 
operational requirements. 
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Navy waivers consist of waivers from compliance with the criteria for 
certification of readiness for operational test and evaluation and waivers for 
deviations from TEMP testing requirements. The program manager requests a 
test waiver in an operational test and evaluation certification message. If a waiver 
request is anticipated, the program manager is to coordinate the waiver with the 
program sponsor; the Test and Evaluation Division, Office of the Director of 
Navy Test and Evaluation and Technology Requirements; and the 
COMOPTEVFOR before an operational test and readiness review. The Director, 
Navy Test and Evaluation and Technology Requirements approves waiver 
requests, as appropriate, in coordination with the COMOPTEVFOR; the Deputy 
Chief of Naval Operations (Logistics); the Deputy Chief of Naval Operations 
(Resources, Warfare Requirements, and Assessments); and the program sponsor. 


The Navy Instruction further states that approval of a test waiver request does not 
alter the test requirement. The program manager is still required to test the 
waived requirement in subsequent operational testing. A waiver may result in 
limitations to the scope of testing that preclude the COMOPTEVFOR from fully 
resolving all COIs. The Office of the COMOPTEVFOR is not to assess waived 
requirements in its analysis to resolve COIs, but may comment on the waived 
requirements in the applicable test report. 


Air Force Policy. Air Force Instruction 99-102 requires initial operational test 
and evaluation to determine the operational effectiveness and suitability of 
systems undergoing research and development efforts. Initial operational test and 
evaluation supports decisions of the milestone decision authorities to authorize 
low-rate initial production and full-rate production, and declaration of initial 
operational capability. Further, the Instruction requires a dedicated phase of 
initial operational test and evaluation for Acquisition Category I and II programs, 
and is strongly recommended for all others. The Instruction also requires the 
initial operational test and evaluation to completely and unambiguously answer 
all critical operational issues as thoroughly as possible, and not defer testing into 
follow-on operational test and evaluation unless unavoidable. Additionally, the 
Instruction requires that all approved operational test and evaluation waivers be 
documented in the TEMP. 


To waive or modify fundamental provisions in operational test and evaluation 
plans, the Instruction requires the system program manager to prepare a request. 
With the user and developer concurrence, the system program manager is to 
submit the request through the Air Force Operational Test and Evaluation Center 
to the Air Force Test and Evaluation Directorate. The system program manager 
and the system’s program element monitor are to document any approved 
operational test and evaluation waivers in the TEMP and the program management 
directive, respectively. 


Air Force Manual 63-119 requires a waived requirement to be tested in 
subsequent operational test and evaluations or the user must change the ORD 
requirement. Before initial operational test and evaluation, the system program 
manager is to list any required waivers or areas excluded from the test and 
evaluation, the rationale, and future plans to clear the waivers, and provide a 
summary of the list to the operational tester. In response, the operational tester is 
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to indicate whether to proceed with the test and is to discuss the effects of test 
limitations and test waivers on resolving operational test issues in the test report. 


Air Force Instruction 99-102 states that the final operational test and evaluation 


report must contain descriptions of test methods and limitations, definitive test 
results, and system capabilities and limitations as measured against the ORD. 
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Appendix D. Description of Major Defense 
Systems Reviewed 


Department of the Army 


Javelin. The Javelin, an Acquisition Category IC program, is a man-portable, 
fire-and-forget, medium anti-tank missile employed by dismounted troops to 
defeat current and future threat armored combat vehicles out to 2,500 meters. The 
Javelin attacks most targets from the top to defeat explosive reactive armor; it 
also has the capability, in the direct fire mode, to attack targets under cover or 
those that would be unreachable by top attack. The Javelin consists of a missile 
in a disposable launch tube and a reusable Command Launch Unit, with a trigger 
mechanism and day/night sighting device for surveillance, target acquisition, and 
built-in-test capabilities. The missile locks onto the target before launch using an 
infrared focal plane array and on-board processing, which also tracks the target 
and guides the missile to the target after launch. A full-up system weighs less 
than 50 pounds. 


The Javelin includes a training system consisting of three devices: the Missile 
Simulation Round, the Basic Skills Trainer, and the Field Tactical Trainer. The 
Missile Simulation Round is a form, fit, and weight, but not functional, 
representation of the missile in its launch tube and is used to familiarize the 
gunner with the physical characteristics of the Javelin. The Basic Skills Trainer is 
used in classrooms to develop the basic tactical and technical gunnery skills to 
operate the Javelin. The Field Tactical Trainer refines the gunner’s abilities and 
enables the gunner to participate in range training and force-on-force exercises. 


Patriot Advanced Capability-3. The Patriot, an Acquisition Category ID 
program, is an air-defense that uses guided missiles to engage and destroy 
air-breathing targets and tactical ballistic missiles. The Patriot Advanced 
Capability-3 is the latest version. The latest version will include a multi-function 
radar, an engagement control station, and communications relay groups for 
communicating with remote launchers and the battalion headquarters. 


Tactical Unmanned Aerial Vehicle. The Tactical Unmanned Aerial Vehicle 
system, an Acquisition Category II program, will provide the ground maneuver 
brigade commander with a day and night reconnaissance, surveillance, target 
acquisition, and battle damage assessment system. The Tactical Unmanned 
Aerial Vehicle provides the commander with enhanced situational awareness, a 
target acquisition capability, the ability to conduct battle damage assessment, and 
enhanced battle management capabilities. Those capabilities will provide the 
commander with dominant situational awareness and allow him to maneuver to 
points of positional advantage with speed and precision to conduct decisive 
operations. The Tactical Unmanned Aerial Vehicle, in conjunction with other 
systems, will provide the tactical commander with information superiority 
contributing to the full-dimensional protection of his force and precision 
engagement of the enemy. 
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Department of the Navy 


Cooperative Engagement Capability. The Cooperative Engagement Capability, 
an Acquisition Category ID program, is a system of hardware and software that 
allows the sharing of radar data on air targets among ships. Radar data from 
individual ships of a Battle Group is transmitted to other ships in the group 
through a line of sight data distribution system. Each ship uses identical data 
processing algorithms resident in its Cooperative Engagement Processor, 
resulting in each ship having essentially the same display of track information on 
aircraft and missiles. Accordingly, an individual ship can launch an anti-air 
missile at a threat aircraft or anti-ship cruise missile within its engagement 
envelope, based on radar data relayed to it by another ship. 


F/A-18E/F Super Hornet. The FA-18E/F Super Hornet, an Acquisition 
Category IC program, is a multi-mission, day/night strike fighter aircraft designed 
to add to the capability and tactical flexibility of the Carrier Air Wing through 
improvements in the FA-18C/D's range, endurance, and carrier bring-back 
payload and weapons payload. The FA-18E/F features a larger airframe with 
more fuel capacity, two additional store stations and increased survivability with 
additional capacity for capability upgrades and growth. This aircraft will also 
serve as an airborne tanker, further improving Battle Group flexibility and 
mobility. 


Joint Stand-Off Weapon. The Joint Stand-Off Weapon, an Acquisition 
Category ID program, is a family of efficient 1,000-pound class, air-to-surface 
glide weapons that provide for low observability, multiple kills per pass, 
preplanned missions, standoff precision engagement, and launch and leave 
capability against a wide range of targets during day/night, all weather conditions. 
The system is employed as a force multiplier in a joint warfare environment for 
interdiction of fixed, relocatable and mobile, light and heavy armored targets, 
massed mobile armored targets, anti-personnel, and air-to-surface threats. The 
system primarily functions in a preplanned mission mode where the system can 
store up to eight targets; however, the system is to allow pilot manual inputs as 
well as third-party targeting. The weapon will be land and carrier based. 


V-22 Osprey. The V-22 Osprey, an Acquisition Category ID program, is a 
tilt-rotor, vertical take-off and landing, multi-mission aircraft developed to fill 
multi-Service combat operational requirements. The MV-22 will replace the 
Marine Corps assault helicopters in the medium lift category (CH-46E and 
CH-53D), contributing to the dominant maneuver of the Marine landing force, as 
well as supporting focused logistics in the days following commencement of an 
amphibious operation. The Air Force requires the CV-22 to provide a long-range 
vertical take-off and landing insertion and extraction capability and to supplement 
the Special Operations Forces MC-130 aircraft in precision engagement. The 
tilt-rotor design combines the vertical flight capabilities of a helicopter with the 
speed and range of a turboprop airplane and permits aerial refueling and 
worldwide self-deployment. 
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Department of the Air Force 


F-15 Tactical Electronic Warfare System. The F-15 Tactical Electronic 
Warfare System Band 1.5 is an Acquisition Category HI program. The 
acquisition program contributes to full-dimensional protection by improving 
individual F-15 aircraft probability of survival through improved air crew 
situation awareness of the radar-guided threat environment, cueing both active 
and passive countermeasures in the Band 1.5 frequency spectrum, and adding a 
feature for jamming optimization against specific threats. The Tactical Electronic 
Warfare System provides electronic detection and identification of surface and 
airborne threats. In addition, it allows for activation of appropriate 
countermeasures, including electronic jamming and dispensing of expendables 
such as chaff and flares. 


Fighter Data Link. The Fighter Data Link, an Acquisition Category ID program, 
provides Link 16 data link networking with other Link 16 capable fighter aircraft, 
command, and control systems to support synchronized operations. Link 16 is a 
data link with a common message standard and robust jam-resistant 
communications waveform providing Joint and Multinational interoperability to 
enable forces to operate effectively together. 


Joint Direct Attack Munition. The Joint Direct Attack Munition, an 
Acquisition Category ID program, is a low cost, autonomously controlled 
guidance kit for the Air Force and Navy 2,000-pound general-purpose bombs and 
the 1,000-pound bomb. The Air Force inventory of bombs will be configured 
with Joint Direct Attack Munition guidance kits and accessories. Actual weapon 
launch occurs when the aircrew has flown the aircraft into the weapon launch 
acceptability region. The launch acceptability region is the three-dimensional 
area in space in which the weapon may be released to fly directly to a selected 
target on a pre-determined bearing. The Joint Direct Attack Munition is designed 
to be employed by a variety of fighter/attack and bomber aircraft, allowing 
precision engagement from all altitudes under adverse environmental conditions. 


Predator Unmanned Aerial Vehicle. The Predator Unmanned Aerial Vehicle 
System, an Acquisition Category II program, is a theater asset that is to provide 
cued and non-cued reconnaissance, surveillance, and targeting capability. The 
Predator system comprises both air and ground segments. The Predator began as 
an advanced concept technology demonstration program that transitioned to an 
Acquisition Category II program. 
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Appendix E. Critical Operational Issues not 
Resolved Because of Approved Test 
Waivers or Test Limitations 


This appendix shows operational effectiveness or operational suitability 
categories and the associated critical operational issues that could not be resolved 
during operational testing before system full-rate production decisions because of 
waivers and other test deferrals for the eight DoD weapon systems affected by 
waivers. 


Department of the Army 


Javelin 


Operational Suitability Issues 


Critical Operational Issue (Reliability). Is the Javelin operationally suitable 
when employing existing doctrine: manpower and personnel integration, 
reliability, availability, and maintainability, and logistic supportability concepts? 
The Javelin entered initial operational test and evaluation with limitations 
associated with reliability. Engineering and manufacturing development testers 
estimated that the field tactical trainer (range) would achieve 25 hours, rather than 
the 100 hours required. The system completed initial operational test and 
evaluation in December 1993, the results of which indicated that the system had 
unresolved operational suitability issues related to the reliability parameters 
prescribed for system maturity; specifically, the training devices were unreliable. 
According to the Deputy Under Secretary of the Army (Operations Research), the 
U.S. Army Infantry reevaluated the requirements in 1994 and reduced the 
100-hour requirement to 50 hours that should be attained at the time of system 
maturity, 3 years after the full-rate production decision. The Deputy also stated 
that the requirement for the full-rate production decision was 32 hours as shown 
in the System Evaluation Report, March 27, 1997. The Army operational testers 
resolved the operational suitability issues after the full-rate production decision. 


Patriot Advanced Capabilities (PAC-3) Program Office 


Operational Effectiveness Issues 


Critical Operational Issue. Does the Patriot Advanced Capability-3 enhance the 
effectiveness of the Patriot system? The testers were not able to resolve the 
operational effectiveness COI because the Army deferred test requirements: 


e for the missile to effectively destroy a target and counteract 
countermeasures, 


e for the ground system to distinguish between targets, and 
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e for the system to operate in all required environments. 


In consideration of performance problems identified during limited user testing, 
the Program Manager obtained the approval of the Joint Requirements Oversight 
Council to defer those test requirements as part of an evolutionary development 
program after the planned full-rate production decision. As a consequence, the 
Army Test and Evaluation Command determined in the limited user tests 
conducted before the LRIP decision that the Patriot Advanced Capability-3 had 
limited operational effectiveness. Because the Joint Requirements Oversight 
Council deferred those test requirements, the operational effectiveness COI will 
not be fully resolved before the program proceeds beyond LRIP. To resolve the 
deferred test requirements, the Program Office stated that the Army increased 
funding for system development by approximately $90 million to demonstrate 
those ORD requirements. During initial operational test and evaluation 
conducted in 2002, the Army Test and Evaluation Command and the Director, 
Operational Test and Evaluation, determined that the Patriot Advanced 
Capability-3 was operationally effective with limitations. 


Operational Suitability Issues 


Critical Operational Issue. Can the PAC-3 be sustained in an operational 
environment? During initial operational test and evaluation conducted in 2002, 
the Army Test and Evaluation Command and DOT&E, determined that the Patriot 
Advanced Capability-3 was operationally suitable with limitations, including 
reliability and availability. 


Reliability — Limited user test revealed a number of PAC-3 system 
reliability problems. The communications relay group mean time between critical 
mission failure rate was seven times greater than allocated, and the engagement 
control station mean time between critical mission failure rate was twice as large 
as allocated. The routing logic radio interface units problems were the primary 
reasons that the fire unit mean time between critical mission failure rate was 
1.7 times greater than the requirement. The routing logic radio interface units and 
data link terminals were major contributors to poor system reliability. The 
“System Evaluation Report of the Patriot Advanced Capability-3 Initial 
Operational Test and Evaluation,” indicated improvements for those issues, 
however, the overall fire unit mean time between critical mission failure of 
19.8 hours was less than threshold requirement of 21 hours. 


Availability - Specifics concerning limitations to availability are 
classified and are not detailed in this report. 


Tactical Unmanned Aerial Vehicle Program Office 
The Tactical Unmanned Aerial Vehicle is a commercial off-the-shelf program 


that did not undergo developmental testing. The Project Office certified the 
system as ready for operational testing, given the following limitations. 
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Operational Effectiveness Issues 


Critical Operational Issues. The Tactical Unmanned Aerial Vehicle limitations 
to testing included two COIs concerning operational effectiveness. 


e Does the Tactical Unmanned Aerial Vehicle system provide the 
commander timely and accurate reconnaissance, surveillance, and 
target acquisition information? 


e Does the Tactical Unmanned Aerial Vehicle system provide adequate 
coverage of the ground maneuver brigade commander’s area of 
interest? 


The Tactical Unmanned Aerial Vehicle program entered initial operational test 
and evaluation with known limitations affecting the resolution of the operational 
effectiveness COIs. For example, the ground control station was not compatible 
with the tactical control system architecture because the tactical control system 
was still in development. As a result, Army testers were not able to determine 
whether interoperability characteristics of the vehicle met the ORD requirements. 
The program manager and the users listed the test limitations in memorandums to 
the Army testers instead of requesting waivers to testing the ORD requirement. 
The Army plans to conduct the deferred test requirements during future 
operational test and evaluation. 


Operational Suitability Issues 


Critical Operational Issue (Logistics Supportability). Does the Tactical 
Unmanned Aerial Vehicle system meet the ground maneuver brigade 
commander’s sustained combat requirements? The Tactical Unmanned Aerial 
Vehicle Program Office deferred planned operational tests that affected the 
resolution of the COI requiring the system to meet the ground maneuver brigade 
commander’s sustained combat requirements. Specifically, the Tactical 
Unmanned Aerial Vehicle Program Manager in a memorandum, “Initial Tactical 
Unmanned Aerial Vehicle System Limitations for Initial Operational Test and 
Evaluation (IOTE),” undated, limited the test and evaluation because testers could 
not verify the supportability of the system when using heavy fuel. The ORD 
objective requires the system to use only heavy fuel; however, the system 
available for testing used motor gasoline, which meets the operational 
requirement threshold, the minimal acceptable value necessary to satisfy the need. 


According to DOT&E, even though the system used motor gasoline, the system 
met the fuel requirement by meeting the threshold; however, the Army is 
addressing supportability issues about being able to support a motor gasoline after 
2005. After the planned full-rate production decision, the Army will conduct a 
follow-on operational test and evaluation to test for the heavy fuel requirement. 
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Department of the Navy 


F/A-18E/F Super Hornet 


Operational Effectiveness Issues 


Critical Operational Issues. The F/A-18E/F Super Hornet waivers included 
five COIs concerning operational effectiveness. 


Airborne Tanker Platform. Will the F/A-18E/F function adequately 
as an airborne tanker platform? 


Air-To-Ground Sensors. Will the F/A-18E/F air-to-ground sensors be 
effective in all mission areas including navigation, target detection, 
target acquisition, target identification, target designation, target track, 
and target surveillance against the current and projected threat in its 
intended operating environment? 


Air-To-Ground Weapons. Will the F/A-18E/F effectively employ 
and support the full capability of all air-to-ground weapons it is 
intended to carry against the current and projected threat in its intended 
operating environment? 


Air-To-Air Sensors. Will the F/A-18E/F effectively employ and 
support the full capability of all air-to-air weapons it is intended to 
carry against the current and projected threat aircraft in its intended 
operating environment? 


Mobility. Will the F/A-18E/F demonstrate adequate mobility to 
conduct wartime operations from ashore or from an aircraft carrier? 


In consideration of performance problems identified during developmental testing, 
the program manager requested the following operational test waivers: 


Lack of provisions for tanker lighting recognition.* 


Excessive time to rebuild and expand mode image after aim-point 
redesignation. 


Inadequate targeting, forward-looking, infrared resolution and 
magnification. 


Targeting, forward-looking, infrared loss of autotrack and scene track 
during maneuvering. 


“Resolved after the full-rate production decision. 
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e Insufficient air-to-ground radar, expand mode resolution. 
e Inadequate targeting, forward-looking, infrared autotrack range. 


e Scaling of radar presentation in “expand 2/3” does not permit precise 
target designation. 


e No laser blanking to outboard weapons stations. 
e Insufficient duration of AIM-9 cooling capacity.’ 


e Unsatisfactory automatic carrier-landing system.* 


Operational Suitability Issues 


Critical Operational Issue (Reliability). Will the reliability of the F/A-18E/F 
support completion of the aircraft’s mission? Based on reliability performance 
problems identified during developmental tests, the Program Executive Officer for 
Tactical Aircraft requested the following operational test waivers: 


e Unacceptable reliability of the fuel and air heat exchanger leak 
detection system.’ 


e Premature failure of the environmental bellows due to misalignment 
during door installation. 


e ALE-50 system requires preflight power-up with weight-on-wheels.* 
e Inadequate secondary power system engine start fault isolation.’ 


e AIM-9M sidewinder captive air training missile roller on assembly 
in-flight failures. 


e Uncommanded map display format changes in one cockpit based on 
map displays selected in other cockpit. 


e Undesirable inter-cockpit display format coupling. 


“Resolved after the full-rate production decision. 


"Deliberative process privilege data omitted. 
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Axle vertical loads encountered during catapult testing exceed 
F/A-18E/F nose landing gear tire and wheel loads qualification limits.* 
Inadequate reliability of the hydraulic drive unit.’ 


Delamination within composite surface layers of the horizontal 
stabilator. 


Engine stalls in maximum afterburner power while aircraft is 
positioned in front of the jet blast deflector.’ 


Engine stalls during catapult stroke.’ 

Engine stator vane retainer rail failures.* 

Unacceptable high failure rate of fuel cells.’ 

Poor reliability characteristics of the fuel/no air valve.* 

High failure rate of the wing fuel cell channel injection plugs.* 
AIM-7 forward lug failure in flight on station two.* 

Pylon loads exceed design limit during maneuvering flight.’ 
Failure of memory unit data to automatically download.’ 


In-flight failure of LAU-127A/A while carrying captive air training 
missile.’ 


Failure of the nose landing gear door forward hinge. 
Failure of the main landing gear outboard door forward hinge.* 
Broken fin retainer springs on the LAU-127A/A launcher. 


Damage and loss of AAR-50 forward looking, infrared coolant access 
door during flight. 


Receiver saturation in engagements within-band forward quarter. 


Critical Operational Issue (Maintainability). Will the F/A-18E/F be 
maintainable by fleet personnel? Based on maintainability problems identified 
during developmental tests, the Program Executive Officer for Tactical Aircraft 
requested the following operational test waivers: 


“Resolved after the full-rate production decision. 
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e Configuration of the hydraulic bay 53 L/R does not provide sufficient 
accessibility to hydraulic tube assemblies, which inhibits proper 
torquing. 


e Inadequate limits for engine high pressure compressor bladed disk 
repair. 


e Wing tip LAU-127A/A damage caused during downloading of AIM-9 
stores after flight.’ 


e Inadequate environmental control system diagnostic capability.* 
e Excessive inspection requirements for AIM-120C on stations 2 and 10.* 
e LAU-127A/A loose screws. 


e Inadequate procedures for repeatable release holdback bar release load 
calibration.* 


Critical Operational Issue (Compatibility). Will the F/A-18E/F be compatible 
with its operating environment? Based on compatibility problems identified 
during developmental testing, the Program Executive Officer for Tactical Aircraft 
requested the following operational test waivers: 


e Poor F/A-18F intercommunication system mechanization.* 


e No provision for reception on one radio while transmitting on the 
other.’ 


e No provision for concurrent transmission (pilot and weapon system 
operator).* 


e ALR-67 V(3) is incompatible with the tactical aircrew combat training 
system.’ 


Critical Operational Issue (Safety). Will the F/A-18E/F be safe to operate and 
maintain? Based on safety problems identified during developmental tests, the 
Program Executive Officer for Tactical Aircraft requested the following 
operational test waivers: 


e Potential for personnel injury due to location of hydraulic fluid level 
indicator. 


e Erroneous velocity vector performance without aid from the global 
positioning system. 


e Inadequate attitude reference indication during night carrier vessel 
approach.* 


“Resolved after the full-rate production decision. 
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Critical Operational Issue (Human Factors). Will the human factors aspects of 
the F/A-18E/F support completion of the aircraft’s mission? The Program 
Executive Officer for Tactical Aircraft requested and the Director of Navy Test 
and Evaluation and Technology Requirements approved a test waiver for the 
ALR-67 power switch on the center pedestal because it was not within reach of the 
pilot while he was strapped in the seat. 


V-22 Osprey 


Operational Effectiveness Issues 


Critical Operational Issue (Assault Support). Will the V-22 demonstrate the 
operational performance necessary to effectively execute assault support 
operations in its intended environment? Because of deficiencies identified during 
developmental testing, the Director of Navy Test and Evaluation and Technology 
Requirements approved the following operational test waivers: 

e Inadequate cargo handling system. 


e Avionics navigation system does not provide data other than World 
Geodetic System. 


e Exterior lighting for formation flight is inadequate. 
e External load interferes with radar altimeter. 
e Lower cabin door operation during hover operations. 
e Unable to fastrope out of cabin door. 
Further, the Director of Navy Test and Evaluation and Technology Requirements 


approved an operational test waiver for maximum short take off (shipboard) 
48,500 pounds, rather than mission profile specific weight because the capability 


“Resolved after the full-rate production decision. 
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had not been tested developmentally before operational evaluation, according to 
the Deputy Assistant Secretary of the Navy (Research, Development, Test and 
Evaluation). 


The Director of Navy Test and Evaluation and Technology Requirements also 
approved an operational test waiver for the defensive weapon system because the 
system was still under development and had not been fielded before the 
operational test and evaluation, according to the Deputy Assistant Secretary of the 
Navy (Research, Development, Test and Evaluation). 


Critical Operational Issue (Self-Deployment). Will the V-22 demonstrate the 
operational performance necessary to effectively self-deploy in its intended 
operating environment? Because of performance problems identified during 
developmental testing, the Director of Navy Test and Evaluation and Technology 
Requirements approved the following operational test waivers: 


e Aircraft was not cleared to refuel from a KC-135 tanker. 
e Aircraft was not cleared for aerial refueling. 


In addition, the Director of Navy Test and Evaluation and Technology 
Requirements approved an operational test waiver for crashworthy auxiliary fuel 
tanks because the system was still under development and had not been fielded 
before the operational test and evaluation, according to the Deputy Assistant 
Secretary of the Navy (Research, Development, Test and Evaluation). 


Critical Operational Issue (Survivability). Will the susceptibility and 
vulnerability characteristics of the V-22 allow the successful completion of its 
mission in its intended operating environment? Because of performance problems 
identified during developmental testing, the Director of Navy Test and Evaluation 
and Technology Requirements approved the following operational test waivers. 


e AN/APR-39A(V)2 degraded Band 2 analysis of alternatives. 
e Defensive weapon system not available for test. 


Further, the Director of Navy Test and Evaluation and Technology Requirements 
approved an operational test waiver for the Aircraft being cleared for air combat 
maneuveringbecause the capability had not been tested developmentally before 
operational evaluation, according to the Deputy Assistant Secretary of the Navy 
(Research, Development, Test and Evaluation). 


In addition, the Director of Navy Test and Evaluation and Technology 
Requirements approved an operational test waiver for crashworthy auxiliary fuel 
tanks because the system was still under development and had not been fielded 
before the operational test and evaluation, according to the Deputy Assistant 
Secretary of the Navy (Research, Development, Test and Evaluation). 
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Operational Suitability Issues 


Critical Operational Issue (Reliability, Availability, and Maintainability). 
Will the V-22 be reliable in its intended operating environment? Will the 
availability of the V-22 support its mission? Will the V-22 be maintainable in its 
intended operating environment? Because of performance problems identified 
during developmental testing, the Director of Navy Test and Evaluation and 
Technology Requirements approved an operational test waiver of the V-22 mean 
time between failure and false alarm rate technical thresholds. 


Critical Operational Issue (Compatibility). Will the V-22 be compatible with 
its operating environment? Because of performance problems identified during 
developmental testing, the Director of Navy Test and Evaluation and Technology 
Requirements approved the following operational test waivers. 


e Inadequate cockpit/cabin nuclear biological and chemical overpressure 
protection. 


e Unable to align Light Weight Inertial Navigation System without 
Global Positioning System signal. 


In addition, the Director of Navy Test and Evaluation and Technology 
Requirements approved an operational test waiver for the aircraft to operate in 
icing conditions because the system was still under development and had not been 
fielded before the operational test and evaluation, according to the Deputy 
Assistant Secretary of the Navy (Research, Development, Test and Evaluation). 


Critical Operational Issue (Safety). Will the V-22 be safe to operate and 
maintain? Because of performance problems identified during developmental 
testing, the Director of Navy Test and Evaluation and Technology Requirements 
approved the following operational test waivers. 

e Excessive force required to disconnect the intercommunication system. 


e Pilot and copilot seats incorporate nonqualified inertial reels. 


e Autorotative descent cannot be maintained while attempting engine air 
start. 


In addition, the Director of Navy Test and Evaluation and Technology 
Requirements approved an operational test waiver for the ground collision 
avoidance and warning system because the system was still under development 


"Deliberative process privilege data omitted. 
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and had not been fielded before the operational test and evaluation, according to 
the Deputy Assistant Secretary of the Navy (Research, Development, Test and 
Evaluation). 


Department of the Air Force 


F-15 Tactical Electronic Warfare System 


Operational Suitability Issues 


Critical Operational Issue (Reliability and Availability). Does the operational 
readiness of the Band 1.5 system support F-15E mission requirements? The 

Air Force will depend on the capability of the built-in-test of the F-15 Tactical 
Electronic Warfare System (the System) to identify faults and take corrective 
action in a timely manner. The May 2000 TEMP did not show any test limitations 
associated with the built-in-test; however, it did show that the System had an open 
deficiency for an excessive built-in-test false alarm rate. Although the deficiency 
existed, the Air Force conducted initial operational test and evaluation in FY 2000 
and determined that the built-in-test false alarm rate was 65 percent, compared to 
the required built-in-test false alarm rate of less than 20 percent. The excessive 
false alarm rate caused numerous unwarranted maintenance actions and distracted 
the air crew’s attention from primary mission functions. Even with the excessive 
false alarm rate, the Air Force approved the system for production because the 
System provided a greater capability, according to DOT&E representatives. In 
addition, DOT&E approved the TEMP and operational test and evaluation plan for 
the Band 1.5 system, both of which addressed the false alarm rate and required 
future testing, according to Air Force Test and Evaluation Directorate 
representatives. The representatives also stated that the Air Force submitted plans 
to DOT&E to correct the false alarm rate problems in the next increment, 

Band 5.0. 


Fighter Data Link 


Operational Suitability Issues 


Critical Operational Issue (Reliability). Is the Fighter Data Link terminal 
suitable when integrated in an F-15? The operational test report included a test 
limitation related to the built-in-test functionality. During initial operational test 
and evaluation, inadequate flight hours were available to evaluate the Fighter Data 
Link built-in-test and logistics supportability. Accordingly, Air Force operational 


"Deliberative process privilege data omitted. 
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testers did not fully test mean time between critical failures, the integrated 
diagnostics, and logistics support. As a result, Air Force operational testers were 
not able to resolve whether the Fighter Data Link terminal was suitable when 
integrated into the F-15. Consequently, Air Force testers deferred further 
evaluation of built-in-test functionality until follow-on operational test and 
evaluation. According to DOT&E, during follow-on operational test and 
evaluation, the Fighter Data Link was able to demonstrate 963 hours even though 
the ORD requires 1,000 hours. 


Predator Unmanned Aerial Vehicle 


Operational Effectiveness Issues 


Critical Operational Issue (Mission Utility). Can the Predator system perform 
reconnaissance, surveillance, and target acquisition tasks to support battle 
management, execution, and operations other than war? Although the Predator 
System Program Office certified that the Predator was ready to enter operational 
testing, it identified operational effectiveness limitations related to the system’s 
ability to recognize tactical-sized targets that were identified during informal 
developmental testing. Based on operational testing conducted before the full-rate 
production decision, DOT&E stated that the Predator was not operationally 
effective, which included the inability to recognize tactical-sized targets 


Operational Suitability Issues 


Critical Operational Issue (Reliability). Does the Predator system meet the 
warfighter’s sustained combat requirements? The System Program Office certified 
the Predator as ready to enter operational test and evaluation with test limitations 
concerning the aircraft’s effective time on station that were identified during 
informal developmental testing. One limitation was the use of pre-production 
wing sets instead of production units during operational testing to determine the 
aircraft’s effective time on station. The ORD requirement for effective time on 
station was 75 percent, but the actual effective time on station recorded during 
testing was about 70 percent. Correcting for the use of pre-production wing sets, 
the Air Force Operational Test and Evaluation Center stated that it was 68 percent 
confident that the system would meet the ORD requirement. However, the 

Air Force Operational Test and Evaluation Center stated that, under sustained 
operations, it was highly probable that air vehicle failures would have a negative 
effect on the aircraft’s effective time on station. 


Critical Operational Issue (Interoperability). Can the Predator system 
successfully accomplish the mission planning, launch and recovery, in-flight 
operations, and interoperability with other command, control, communication, and 
computers and intelligence systems necessary to perform its missions? The System 
Program Office exempted planned operational tests of interoperability data links 
and file formats because the Joint Interoperability Test Command had not certified 
the system for interoperability, thereby affecting the resolution of the COI. 
Accordingly, the System Program Office certified that the Predator Unmanned 
Aerial Vehicle was ready for operational testing with a stipulation that, if the Joint 
Interoperability Test Command did not certify the system for 
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interoperability during initial operational test and evaluation, the System Program 
Office would complete the certification process after the completion of operational 
testing. According to DOT&E FY 2001 Annual Report, the Joint Interoperability 
Test Command had not yet certified the Predator for interoperability because of the 
untested critical requirements. Further, the Joint Interoperability Test Command 
had certified only three of the seven critical system interfaces because the 
remaining interfaces were not available for evaluation. The System Program 
Office deferred further interoperability testing of the Predator until the Military 
Departments field ground stations that are equipped with tactical control stations. 
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Appendix F. Audit Response to the Director, 
Operational Test and Evaluation 
Comments on the Report 


The detailed responses to the comments from the Director, Operational Test and 
Evaluation on statements in the draft report follow. The DOT&E commented on 
limiting the use of waivers, live-fire test and evaluation, and the Fighter Data 
Link. The complete text of those comments is in the Management Comments 
section of this report. 


Limiting the Use of Waivers. The Director stated that his office seeks to limit 
the issuance of waivers and eliminate their interference with evaluations of 
operational effectiveness and suitability. 


Live-Fire Test and Evaluation. The Director stated that the Background section 


of the report states that, “...evaluation and when live-five test and evaluation...,” 
and that it should be revised to state “...evaluation and when live-fire test and 
evaluation....” 


Audit Response. We revised the report as suggested. 


Fighter Data Link. The Director stated that, during initial operational test and 
evaluation, inadequate flight hours were available to evaluate the Fighter Data 
Link built-in-test and logistics supportability. Further, he stated that although the 
ORD requires 1,000 hours, during follow-on operational test and evaluation, the 
Fighter Data Link was able to demonstrate 963 hours. 


Audit Response. We revised the report as suggested. 
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Appendix G. Audit Response to Army Comments 
on the Report 


The detailed responses to the comments from the Deputy Under Secretary of the 
Army (Operations Research) on statements in the draft report follow. The 
complete text of those comments is in the Management Comments section of this 
report. 


Management Comments on the Background and Appendix C 
and Audit Response 


The Deputy commented on the Background section and Appendix C, “Test and 
Evaluation Policy,” of the report. 


Background. The Deputy stated that the report states that we initiated the audit to 
address the DOT&E concern about programs beginning operational test and 
evaluation without completing sufficient developmental test and evaluation. 
However, the Deputy further stated that we limited the scope of the audit to the test 
waiver and deferral process, which he believed was not sufficiently broad enough 
to yield findings or recommendations that could improve the robustness of 
developmental testing or the likelihood of conducting a successful operational test. 


Audit Response. We limited the scope of the audit to the test waiver and deferral 
process based on concerns DOT&E expressed in his FY 2000 annual report. 
Specifically, the Director expressed concern about programs beginning operational 
testing without completing sufficient developmental test and evaluation and about 
the Navy’s use of test waivers. The primary focus of the audit was not 
developmental testing or the likelihood of conducting a successful operational test 
because, as the Director stated, the reason that developmental testing was not 
robust was that program managers have reduced developmental testing to save 
time and money, thereby postponing the problems to operational testing. 
Therefore, we focused the audit on the Director’s concern about the Military 
Departments’ use of test waivers. 


Appendix C. The Deputy recommended that the report mention the COI approval 
authority for each of the Military Departments. The Deputy rationalized that, 
because COIs are important to the decision makers, mentioning the approval 
authority would reveal how the decision makers are considering COIs and COI 
criteria as tester or decision maker products. He also stated that, in accordance 
with Army Regulation 73-1, “Test and Evaluation Policy,” COI criterion approval 
authority is based on the type of program. For example, the approval authority for 
material and tactical programs with command, control, communications, 
computers, and intelligence and information technology requirements would be the 
Deputy Chief of Staff for Programs (G-8) and for nontactical programs with 
similar requirements, it would be the Army Chief Information Officer. Further, the 
Deputy stated that Army test and evaluation policy addresses a mandatory 
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attachment, “Requirements/Test Crosswalk Matrix,” which identifies the source of 
each operational requirement agreed upon with the TEMP approval. 


Audit Response. The primary focus of the audit was not on how the decision 
makers consider COIs and COI criteria for approval, but rather on the waiver of 
the testing for the COIs and the need for subsequent testing of those COIs to 
ensure that the system is operationally effective and suitable. 


Management Comments on Finding A and Audit Response 


The Deputy discussed operational effectiveness and operational suitability issues 
associated with the Javelin, the Patriot Advanced Capability-3, and the Tactical 
Unmanned Aerial Vehicle programs. 


Javelin. For the Javelin Program, the Deputy commented on unresolved issues, 
the table of COIs, operational requirements, suitability issues, training category, 
and the description of the Javelin. 


Unresolved Issues. The Deputy recommended removing the discussion 
about the Javelin from the report because it did not support the report’s overall 
argument that unresolved issues caused costly retrofits of fielded units or an initial 
operational performance that is less than expected to defeat the threat. 


Audit Response. The System Evaluation Report, March 27, 1997, states 
that all reliability, availability, and maintainability requirements for the full-rate 
production decision were met with the exception of the field tactical trainer (range) 
mean time between operational mission failures. Therefore, the critical operational 
issue for reliability was not fully resolved before the full-rate production decision, 
thereby resulting in initial operational performance that was less than expected and 
the potential for fixes that will result in retrofit costs for fielded units. 


Table of COIs. The Deputy disagreed with the issue, suggesting that the 
table, “Critical Operational Issues Not Fully Resolved Because of Approved Test 
Waivers or Test Limitations for the Eight Defense Systems Affected,” be changed 
to show that the Javelin did not have unresolved issues for reliability. 


Audit Response. The Javelin did have an unresolved reliability issue 
when the full-rate production decision was made. Specifically, the System 
Evaluation Report states that the field tactical trainer (range) attained a mean time 
between operational mission failures of 25.9 hours that did not meet the threshold 
requirement of 32 hours. 


Operational Requirements. The Deputy disagreed with the issue, stating 
that the requirement for a 100-hour mean time between operational mission 
failures for the field tactical trainer (range) cited in the report was a requirement 
written in the Joint Service Operational Requirements before the tactical and 
training systems had a firm hardware configuration. Further, he stated that the 
U.S. Army Infantry reevaluated the requirements in 1994 and reduced the 
100-hour requirement to 50 hours that should be attained at the time of system 
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maturity, 3 years after the full-rate production decision. The Deputy also stated 
that the requirement for the full-rate production decision was 32 hours as shown in 
the System Evaluation Report, March 27, 1997. In addition, he stated that, 
although the 25-hour mean time between operational mission failures did not meet 
the 32-hour requirement, the difference between 25 hours and 32 hours was closer 
than between 25 hours and 100 hours and would not have caused the Army to not 
field the Javelin. The Deputy stated that the majority of the problems with the 
field tactical trainer (range) and the associated fixes were simple and effective as 
indicated in the Army Test and Evaluation Command assessment, “System 
Assessment of Javelin Reliability and Availability Maturity,” December 2000. 


Audit Response. Based on the Army comments, we revised the report to 
address the 32-hour requirement for the full-rate production decision and the 
50-hour requirement for system maturity. The Army comments affirm the report 
statement that the attained mean time between operational mission failures for the 
field tactical trainer (range) did not meet the threshold requirement. 


Suitability Issues. The Deputy disagreed with the issues, stating that the 
report incorrectly states that the Army operational testers resolved the field tactical 
trainer (range) suitability issues after the full-rate production decision. He did not 
believe that the report was correct because the Army Test and Evaluation 
Command conducted a limited user test from March through May 1996, which 
resolved those suitability issues. The Deputy stated that after the limited user test, 
the Army Test and Evaluation Command conducted a customer confirmatory test. 
Further, he stated that the program successfully completed the customer 
confirmatory test that demonstrated the suitability of the field tactical trainer, after 
which the Army System Acquisition Review Council made its full-rate production 
decision in July 1997. 


Audit Response. When asked, the Deputy’s representative was unable to 
provide support to validate the statement that the customer confirmatory test 
successfully demonstrated the suitability of the field tactical trainer (range). As 
stated in the report, the System Evaluation Report states that all reliability, 
availability, and maintainability requirements for the full-rate production decision 
were met with the exception of the field tactical trainer (range) mean time between 
operational mission failures. Therefore, the COI for reliability was not fully 
resolved before the full-rate production decision. 


Training Category. The Deputy suggested adding the training category to 
the table in the report addressing COIs that were not fully resolved. 


Audit Response. According to the Deputy’s representative, the Deputy 
wanted to add the training category to the table because he believed that the field 
tactical trainer (range) deficiencies were training rather than reliability issues. We 
did not add the training category to the table because the System Evaluation 
Report indicates that the mean time between operational mission failures 
requirement for the field tactical trainer (range) was a reliability issue. 


Description of the Javelin. The Deputy suggested that the description of 


the Javelin include the description of the field tactical trainer, the only unresolved 
issue with the Javelin. 
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Audit Response. We revised the report as suggested. 


Patriot Advanced Capability-3. For the Patriot Advanced Capability-3 Program, 
the Deputy commented on the ORD cancellation, delays in software development 
and missile availability, the operational effectiveness COI, and system reliability 
problems. 


ORD Cancellation. The Deputy stated that the Secretary of Defense 
retroactively canceled all ORDs for missile defense systems in January 2002 and 
required that the replacement ORDs be capabilities based. 


Audit Response. As stated, the Secretary of Defense canceled all ORDs 
for missile defense programs in January 2002; however, the August 2002 TEMP 
for the Patriot Advanced Capability-3 indicates that the testers initiated and 
conducted the testing described in the TEMP based on the 1998 ORD. The TEMP 
further stated that all future testing on the Patriot Advanced Capability-3 would 
adhere to a capabilities-based ORD. According to Patriot Advanced Capability-3 
representatives, the Army combat developers had not yet established a 
capabilities-based ORD for the Patriot Advanced Capability-3. 


Delays in Software Development and Missile Availability. The Deputy 
disagreed with the operational effectiveness issue, stating that, based on delays in 
software development and Patriot Advanced Capability-3 missile availability, the 
Army made a decision to split the system testing, evaluation, and fielding into two 
parts: the First Unit Equipped-Ground (the Ground) and the First Unit 
Equipped-Missile (the Missile). Further, he stated that the Army testers conducted 
a limited user test of the Ground part from March through October 2000. The 
Deputy stated that the test incident reports for the limited user test were 
subsequently resolved or deferred before the Army testers conducted the initial 
operational test and evaluation on the Missile part from January through May 
2002. The Deputy commented that even though the Army deferred testing of some 
of the requirements in the canceled ORD, the user requirements were still valid 
and would be tested and evaluated later. 


Audit Response. The primary focus of this report is the waiver of the 
testing for the COIs and the need for subsequent testing of those COIs to ensure 
that the system is operationally effective and suitable. Because the Patriot 
Advanced Capability-3 missile encountered the deficiencies before the initial 
operational testing and the Army made the decision to defer the resolution of some 
of those deficiencies until after the full-rate production, the program most likely 
will experience retrofit costs for fielded units and initial operational performance 
that may be less than required to defeat the expected threat. 


Operational Effectiveness COI. The Deputy agreed with the statement in 
the draft report that, “The testers were not able to resolve the operational 
effectiveness COI before the planned full-rate production decision because the 
Army deferred test requirements....” He stated that the program had deferrals of 
test requirements that were driven by deferrals of system performance 
requirements and that the Army testers believed that deferrals of test requirements 
was a prudent decision with acceptable risks based on world events. The Deputy 
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concluded that the events in the Mid-East validated the Army tester’s decision to 
defer the test requirements. 


System Reliability Problems. The Deputy disagreed with the system 
reliability issue, stating that the draft report stated that the limited user test 
revealed a number of Patriot Advanced Capability-3 reliability problems involving 
the mean time between critical mission failure rate for the communication relay 
group, the engagement control station, and the fire unit. He stated that difficulties 
with the routing logic radio interface unit was the primary cause of the problems 
involving the mean time between critical, mission failure rate. The Deputy also 
stated that the Lower Tier Project Office’ and Raytheon corrected the difficulties 
with the routing logic radio interface unit after the Ground limited user test and 
before the Missile initial operational test and evaluation. As a result of the 
correction, he stated that the mean time between critical mission failures for the 
fire unit went from 12.6 hours during the limited user test to 19.4 hours® during the 
initial operational test. 


Audit Response. As the Deputy indicated, the “System Evaluation Report 
of the Patriot Advanced Capability-3 Initial Operational Test and Evaluation” did 
indicate that the mean time between critical mission failure rate for the 
communications relay group, the engagement control station, and the fire unit had 
improved. That improvement to 19.8 hours, however, was less than the threshold 
requirement of 21 hours. Accordingly, the System Evaluation Report indicated 
that the system was operationally suitable with limitations, including reliability 
and availability. We revised our report to include the System Evaluation Report 
results that were not available when the draft report was issued. 


Tactical Unmanned Aerial Vehicle. For the Tactical Unmanned Aerial Vehicle 
Program, the Deputy commented on operational effectiveness and operational 
suitability. 


Operational Effectiveness. The Deputy agreed with the operational 
effectiveness issues associated with the ground control station and the tactical 
control system compatibility discussion in the report. 


Operational Suitability. The Deputy disagreed with the operational 
suitability issue, recommending that we delete the logistics supportability issues 
because the Tactical Unmanned Aerial Vehicle met the threshold fuel 
requirements during the May 2002 initial operational test. Although the Army 
approved a system limitation memorandum, he stated that it should be disregarded 
because it references an inconsistent program support requirement within the 
ORD. The Deputy also stated that the current TEMP for the Tactical Unmanned 
Aerial Vehicle includes a limited user test to reevaluate the system after the 


°The Lower Tier Project Office is the project office for the Patriot Advanced Capability-3 Program. 
°The System Evaluation Report shows that this was actually 19.8 hours 
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Tactical Unmanned Aerial Vehicle Program Office develops and integrates a 
replacement heavy fuel engine. 


Audit Response. The ORD objective requires the system to use only 
heavy fuel; however, the system available for testing during initial operational test 
used motor gasoline, which meets the operational requirement threshold, the 
minimal acceptable value necessary to satisfy the need. The ORD also states that 
motor gasoline will not be available after 2005 in the Army inventory and that, 
although motor gasoline is generally available on the local economy, heavy fuel is 
the Army tactical fuel. To integrate a replacement heavy fuel engine into the 
Tactical Unmanned Aerial Vehicle will potentially require costly retrofit as 
addressed in the report finding. As a result, the Army is addressing the issue of 
supporting a motor gasoline system in the future. 


Management Comments on Finding B and Audit Response 


OSD Guidance on Test Deferrals. For the report paragraph, “OSD Guidance on 
Test Deferrals,” the Deputy recommended that we revise the wording in the 
sentence that reads, “Further, the DOT&E issued guidance on August 3, 2001, that 
directed Military Department testers not to issue waivers or deferrals that would 
prevent a complete evaluation of operational requirements.” The Deputy stated 
that Army test and evaluation policy does not authorize testers to waive 
documented test requirements. 


Audit Response. Although the Army test and evaluation policy does not 
authorize testers to waive documented test requirements, the Deputy needs to 
revise its guidance to reference or conform to DOT&E guidance on overall 
waivers. 
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Appendix H. Audit Response to Navy Comments 
on the Report 


The detailed responses to the comments from the Deputy Assistant Secretary of the 
Navy (Research, Development, Test and Evaluation) on statements in the draft 
report follow. The Deputy stated that his comments contained information 
considered privileged in litigation, primarily under the deliberative process 
privilege and, therefore, was exempt from mandatory disclosure under the 
Freedom of Information Act. Where applicable, he marked his comments “For 
Official Use Only.” The complete text of those comments is in the Management 
Comments section of this report. 


Management Comments on the Overall Report and Audit 
Response 


The Deputy commented on OSD test deferral guidance; the potential effects of 
waivers; effects of waivers; report inaccuracies; the acronym for Chief of Naval 
Operations; the acronym for Commander, Operational Test and Evaluation Force; 
types of waivers; and Navy test waivers. 


OSD Guidance on Test Deferrals. The Deputy stated that the report incorrectly 
states that the DOT&E memorandum, August 3, 2001, represents DoD policy on 
test waivers and deferrals. Further, he stated that written policy guidance at the 
DoD level did not exist, that the memorandum was not a directive to the Military 
Departments, and that OSD never implemented the memorandum in subsequent 
DoD directives. The Deputy also stated that each Military Department is within its 
authority to establish and execute their policies regarding test waivers and 
deferrals. Further he stated that the Navy consults with DOT&E on all major 
Defense acquisition programs and DOTS&E oversight programs before considering 
test waivers or deferrals. 


Audit Response. The DOT&E memorandum, August 3, 2001, does represent 
DoD policy on test waivers and deferrals. The DOT&E has the authority to issue 
guidance to the Military Departments on test policy. Section 139, title 10, United 
States Code states that the DOT&E is the principal DoD adviser to the Secretary of 
Defense and the Under Secretary of Defense for Acquisition, Technology, and 
Logistics on operational test and evaluation and the principal operational test and 
evaluation official within the DoD senior management. Section 139 further states 
that the DOT&E will prescribe, by authority of the Secretary of Defense, policies 
and procedures for the conduct of operational test and evaluation in the DoD. 


Although the revised DoD Instruction 5000.2 does not use the term waivers 
discussed in the DOT&E memorandum, it does discuss those deficiencies 
encountered in testing before the LRIP decision for which program managers used 
test waivers, deferrals, and limitations to address. The Instruction requires that 
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those deficiencies be resolved before the system can proceed beyond LRIP and 
that any fixes to those deficiencies be verified during follow-on operational test 
and evaluation. 


Further, the DOT&E stated that the Military Departments should already have 
promulgated the memorandum’s test policy to their operational test organizations. 
The Navy stated that it consults with DOT&E on all major Defense acquisition 
programs and DOT&E oversight programs before considering test waivers or 
deferrals; therefore, it should already have implemented the guidance in the 
DOT&E memorandum. 


Potential Effects of Waivers. The Deputy stated that the report incorrectly states 
that test waivers or deferrals create the inability to resolve COIs needed to assess a 
program’s ability to meet its operational requirements in effectiveness or 
suitability. He stated that before the Navy considers a test waiver or deferral, it 
considers the effect of the test waiver or deferral on the program’s operational 
effectiveness and suitability. Further, the Deputy stated that test waivers and 
deferrals are seldom a factor in COI resolution or determinations of effectiveness 
or suitability. He also stated that, when specified by the Navy Acquisition 
Executive or the milestone decision authority, Navy testers conduct tests to 
provide information to them about a program. The Deputy stated that the Navy 
formally reviews approval of test waivers or deferrals and that the Navy 
Acquisition Executive and the milestone decision authority carefully consider any 
fielding decision. 


Audit Response. As evidenced in the report, the Navy’s consideration of the 
effect of the test waivers and deferrals on the program’s operational effectiveness 
and suitability before approval of the test waiver or deferral did not ensure that 
Navy operational testers could resolve COIs needed to assess a program’s ability 
to meet its operational requirements in effectiveness or suitability. Test waivers 
and deferrals were a factor limiting the Navy operational testers’ ability to 
determine the operational effectiveness and suitability of Navy systems. 


For example, the V-22 Program Office’s request for test waivers states that the 
limitations will affect the complete resolution of the affected COIs for operational 
effectiveness and suitability. Although the F-18E/F Program Office’s request for 
test waivers indicated that the limitations would not affect the resolution of 
affected COIs for operational effectiveness and suitability, the request did list the 
potential effects of the limitations on fleet use. The potential effects on fleet use 
could reasonably be considered the potential effects on operational testing and the 
resolution of affected COIs if those tests were conducted. In accordance with Navy 
policy, however, Navy operational testers were not to use those effects in their 
analysis to resolve COIs. 


The Navy’s policy states that the Office of the COMOPTEVFOR was not to assess 
waived requirements in its analysis to resolve COIs, but could comment on the 
waived requirements in the applicable test report. Without this analysis of the 
effects of the waivers on resolution of COIs, the Navy Acquisition Executive and 
other Navy milestone decision authorities were not being provided with complete 
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information on whether the waived or deferred test requirement would have an 
adverse effect on the system’s operational effectiveness, suitability, or both. With 
incomplete information, the milestone decision authority could make a decision 
that would later result in costly retrofits to fielded units and an initial operational 
performance that may be less than required to defeat the expected threat. 


Effects of Waivers. The Deputy provided comments similar to those he made 
concerning the discussion, “Potential Impacts of Waivers.” In addition to those 
comments, he stated that the report provides no evidence that retrofits occurred for 
Navy programs as a result of test waivers or deferrals. The Deputy stated that the 
two Navy programs discussed in the report, the F/A-18E/F Super Hornet and the 
V-22 Osprey, have not been retrofitted and the Navy does not consider them 
deficient against the expected threat as a result of test waivers or deferrals. 


Audit Response. Although the Deputy stated that the F/A-18E/F Super Hornet 
and the V-22 Osprey have not had any retrofits to date, the V-22 and the 
F/A-18E/F waiver requests indicate that the waived requirements will not be fixed 
and available for testing until after the full-rate production decisions. In his 
response, the Deputy did not indicate when fixes resulting from testing of the 
waived requirements would be incorporated into the full-rate production aircraft 
produced. Accordingly, the extent of retrofit costs will not be known until the 
Navy begins incorporating fixes for unresolved COIs in production aircraft already 
produced. 


Acronyms for Chief of Naval Operations and Commander, Operational Test 
and Evaluation Force. The Deputy stated that throughout the report, Director, 
Test and Evaluation and Technology Requirements, Office of the Chief of Naval 
Operations, and Commander, Operational Test and Evaluation Force are 
unnecessarily referred to in long title and suggested that the report use the long 
title once early in the report and thereafter use CNO (N091) and 
COMOPTEVFOR, respectively. 


Audit Response. Based on IG DoD policy, too many acronyms make a report 
difficult to read and diminish the report’s effectiveness. In the report, we use the 
title, “Director Test and Evaluation and Technology, Office of the Chief of Naval 
Operational,” infrequently; therefore, we did not use the acronym. Because we 
used the title “Commander, Operational Test and Evaluation Force” more often, 
we revised the report to use the acronym as suggested. 


Types of Waivers. The Deputy stated that the report indiscriminately used the 
word “waiver” even though Navy policy defines two types of waivers with 
different ramifications. Specifically, the Navy policy identifies test waivers from 
operational test certification requirements and test waivers from TEMP 
requirements. He stated that the assertions and conclusions in the report could be 
affected by improper, indiscriminate, and inappropriate use of the word “waiver” 
when describing testing of Navy programs. The Deputy recommended that we 
review all Navy sections of the report to identify whether the approved waiver 
was a test waiver from operational test certification requirements (Part A) or test 
waiver from TEMP requirements (Part B). 
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Audit Response. The Office of DOT&E defines a test waiver as deferral of a 
testing requirement that does not eliminate the requirement for testing. Although 
the Navy’s waiver request for the V-22 Osprey and the F/A-18E/F designate the 
deficiencies as either Part A or Part B waivers, the testing requirement for each 
type of deficiency will not be met until after the full-rate production decision. 
Therefore, the test waivers, whether Part A or Part B, are deferrals of testing 
requirements and do not eliminate the requirement for testing as presented in the 
report. 


Navy Test Waivers. The Deputy stated that the definition of a test waiver, as 
stated in the Background section of the report, is vague and inaccurate and does 
not correspond with the definition in Secretary of the Navy Instruction 5000.2B, 
“Implementation of Mandatory Procedures for Major and Non-Major Defense 
Acquisition Programs and Major and Non-Major Information Technology 
Acquisition Programs,” December 6, 1996. He stated that the report needs to 
properly define test-waiver terminology and recommended replacing the 
paragraph, “Navy Test Waiver,” with text that defines the two types of waivers as 
follows: 


e Waivers from compliance with the criteria for certification to 
commence operational testing, referred to as Part A waivers, and 


e Waivers for deviations from the testing requirements directed by the 
TEMP, referred to as Part B Waivers. Those waivers are deferrals that 
are meant to appropriately delay planned testing from one test period 
to a later test period. Further, those deferrals require impact 
statements and a defined follow-on test period. 


Audit Response. The Background section of the report states that, “The Navy 
uses the term ‘waiver’ to mean either deferral of testing of an operational 
requirement or authorization to proceed with testing, even though the program has 
not met the requirements of Secretary of the Navy Instruction 5000.2B.” The 
statements in the Background section of the report about the Navy use of test 
waivers is a synopsis of the test waiver terminology in Secretary of the Navy 
Instruction 5000.2B. The report discusses the Navy’s test waiver policy in further 
detail in Finding B and Appendix C, “Test and Evaluation Policy.” 


Management Comments on Finding A and Audit Response 


The Deputy commented on the resolution of COIs, operational effectiveness 
COIs, operational effectiveness for the F/A-18E/F Super Hornet, Chief of Naval 
Operations involvement, a previous IG DoD Report, a V-22 safety COI, 
F/A-18E/F Super Hornet human factors COI, and the effect of not meeting 
operational requirements. 


Resolution of COIs. 


“Deliberative process privilege data omitted. 
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Audit Response. 


Operational Effectiveness COIs. The Deputy stated that the table, “COIs Not 
Fully Resolved Because of Approved Test Waivers of Test Limitations for the 
Eight Defense Systems Affected,” has only one column for the operational 
effectiveness COIs, while the operational suitability COIs are individually broken 
out. Further, he stated that having one column for the operational effectiveness 
COIs was misleading and implied that effectiveness as a whole was waived or 
deferred. The Deputy suggested that the report list the waived effectiveness 
COIs. 


Audit Response. We did not break out separate categories for operational 
effectiveness COIs because the eight programs did not have common operational 
effectiveness COIs among the Military Departments. The Military Departments 
did have common COIs for operational suitability parameters. Also, as shown in 
Appendix E, the COIs for suitability parameters were more often affected by 
waivers or deferrals of operational testing. In Appendix E, we do provide an 
individual listing of the operational effectiveness COIs that could not be fully 
resolved during operational testing before full-rate production decisions. 


Operational Effectiveness for the F/A-18E/F Super Hornet. 


* 
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Audit Response. 


Chief of Naval Operations Involvement. The Deputy stated that the report 
states that the Director of Navy Test and Evaluation and Technology 
Requirements considered three of the five COIs for operational effectiveness to be 
satisfactorily resolved for the F/A-18E/F Super Hornet. He further stated that this 
statement is technically incorrect because COMOPTEVFOR, not the Director of 
Navy Test and Evaluation and Technology Requirements, resolves COIs. 


Audit Response. We revised the report accordingly. 


Previous IG DoD Report. The Deputy stated that the report cites a previous 

IG DoD report that discusses a limitation to operational testing in a 
COMOPTEVFOR test plan because of the known nonavailability of a 
reconnaissance system for the F/A-18E/F. Further, he stated that the phased 
development of the Shared Reconnaissance Pod, a limitation to testing that 
precluded evaluation of the reconnaissance COI, was well understood before the 
F/A-18E/F operational evaluation. The Deputy also stated that DOT&E, in the 
beyond LRIP report on the F/A-18E/F operational evaluation, stated that “The 
tactical reconnaissance sensor suite intended for eventual employment on the 
FA-18E/F was not developed/cleared for employment during OPEVAL 
[operational evaluation]. The TEMP for the F/A-18E/F, as approved by DOT&E 
in 1992, explicitly stated this limitation with the intention of evaluating this 
capability during a future period of Follow-On Operational Test and Evaluation 
(FOT&E), OT-HI.” Consequently, the Deputy recommended that we remove 
from the report the paragraph citing a previous IG DoD report because it is not 
germane to test waivers and deferrals. He also stated that limitations to the scope 
of operational testing before the operational test and evaluation were not test 
waivers or deferrals. 


Audit Response. We did not remove the subject paragraph from the report. The 
Deputy’s comment confirmed that the known nonavailability of a reconnaissance 
system for the F/A-18E/F deferred the testing of the sub-system until after the 
full-rate production decision for the F/A-18E/F aircraft. Therefore, the previous 
IG DoD Report is germane to the discussion of Navy waivers and deferrals. 


V-22 Safety COI. 


"Deliberative process privilege data omitted. 
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Audit Response. 


F/A-18E/F Super Hornet Human Factors COI. 


Audit Response. 


Effect of Not Meeting Operational Requirements. The Deputy reiterated the 
following statements that we made in the finding. 


Without the Military Departments fully resolving COIs for weapon 
systems during initial operational test and evaluation, weapon systems 
that were not fully ready continued into production, thereby creating a 
potential need for costly retrofit of fielded units and an _ initial 
operational performance that may be less than required to defeat the 
expected threat. 


The Deputy believed that the paragraph conflicted with the DOT&E Beyond 
LRIP Report conclusions regarding F/A-18E/F readiness for full-rate production. 
The Director provided the DOT&E Beyond LRIP Report conclusion as follows: 


The limitations of OPEVAL [operational evaluation] did not inhibit the 
operational evaluation of this aircraft. OPEVAL, as planned and 
conducted provided a test of the operational effectiveness and 
operational suitability of the F/A-18E/F that is adequate for resolution 
of the critical issues and for informing the acquisition decision 
authority regarding the full-rate production decision. 


The Deputy stated that the report did not present any specific evidence that the 
V-22 or F/A-18E/F has or will require costly retrofit to defeat the expected threat. 
He recommended that the V-22 and F/A-18E/F be excluded from the above 
paragraph in the report. 


Audit Response. The V-22 and F/A-18E/F were not excluded from the above 
paragraph because the Navy waived test requirements that did affect the 


resolution of operational effectiveness and suitability COIs. Although approval 
of the test waiver request did not alter the test requirement, Secretary of the Navy 
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Instruction 5000.2B precluded COMOPTEVFOR from assessing the effects of 
waived requirements in its analysis concerning the resolution of COIs. 


For the V-22, the request for test waivers specifically stated that the limitations 
would affect the complete resolution of the affected COIs for operational 
effectiveness and suitability. As a result of Navy policy concerning the resolution 
of COIs, COMOPTEVFOR did not assess the potential effects to fleet use, 
operational testing, and the affected COIs in the analysis. 


Although the F/A-18E/F request for test waivers indicated that the limitations 
would not affect the resolution of affected COIs for operational effectiveness and 
suitability, it listed potential effects on fleet use. In actuality, the potential effects 
on fleet use could reasonably be considered the potential effects on the 
completion of adequate operational testing and the resolution of affected COIs if 
those tests were conducted. In accordance with Navy policy, COMOPTEVFOR 
could not assess those effects in its analysis concerning the resolution of COIs. 


Waiver requests for the V-22 and the F/A-18E/F also indicated that the waived 
items would not be available for testing until after the full-rate production 
decision. 


Management Comments on Finding B and Audit Response 


Management Comments. The Deputy stated that the paragraph, “Navy 
Procedures,” and the accompanying subparagraphs, “Waiver Process,” and 
“Subsequent Operational Tests,” did not accurately define or explain the Navy 
test waiver and deferral policy. He further stated that the paragraph provided an 
inaccurate and incomplete definition of Navy waiver policy and process that 
affects all other Navy portions of the report. The Deputy recommended deleting 
the paragraph and subparagraphs and provided the following replacement text: 


Navy Policy and Procedures. At the time of the V-22 and F/A-18E/F 
OPEVALs [operational evaluations], waivers were governed by 
Secretary of the Navy Instruction 5000.2B, ‘Implementation of 
Mandatory Procedures for major and Non-Major Defense Acquisition 
Programs and Major and Non-Major Information Technology 
Acquisition Programs’ effective 6 December 1996. Two types of 
waivers were defined: 


1. Part A Waivers from compliance with the criteria for OT 
[operational test] certification. These waivers are meant to allow a 
system to enter OPEVAL or FOT&E [follow-on operational test and 
evaluation] even though all criteria have not been met. Waivers do not 
change or delay any system or testing requirements, only the 
data/maturity required by the Decision (DA) to allow a system to enter 
into an Operational Test (OT) period. Data for any waived OT 
certification requirement may be used in COMOPTEVFOR’s final 
analysis to resolve COIs, determine system operational effectiveness, 
operational suitability, and any recommendation regarding fleet 
introduction. 
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2. Part B Waivers. These are deferrals for deviations from the 
testing requirements directed by the Test and Evaluation Master Plan 
(TEMP). These deferrals are meant to appropriately delay planned 
testing from one test period to a later decided-upon test period. This 
moves a requirement from one test period to a later period. A deferral 
may result in limitations to the scope of testing that may preclude 
COMOPTEVFOR from fully resolving all COIs. Deferred items shall 
not be used in COMOPTEVFOR’s final analysis to resolve COIs, but 
maybe commented on in the appropriate sections of the test report. 


Waiver Process. The program managers (PM) or Program 
Executive Officers (PEOs) formulate waiver requests prior to the OT 
readiness and certification review (OTRR) process. When requesting a 
waiver or deferral, the PM shall outline the limitations that the deferral 
or waiver will place upon the system under test, and their potential 
impacts on fleet use. Further, a statement shall be made in the OT&E 
[operational test and evaluation] certification message noting when 
approved deferrals will be available for subsequent operational testing. 
The Director, Navy Test and Evaluation and Technology Requirements 
(CNO(N091)) adjudicates all waiver requests in coordination with 
COMOPTEVFOR, DCNOs [Deputy Chief of Naval Operations] for 
Logistics, _DCNO (Resources, Warfare Requirements, and 
Assessments), and the program sponsor. N091 approves waivers as 
appropriate, and advises OSD (DOT&E) in the case of MDAPs [major 
Defense acquisition programs].” 


Audit Response. The Deputy’s comments on Navy policy and procedures did 
not accurately present information contained in the Secretary of the Navy 
Instruction 5000.2B. Specifically, the Instruction does not allow 
COMOPTEVFOR, for Part A Waivers, to use data for any waived operational test 
in its final analysis to resolve COIs to determine system operational effectiveness 
and operational suitability, and to make any recommendations regarding fleet 
introduction. Further, the Instruction does not state that COMOPTEVFOR, for 
Part B waivers, may comment on deferred items in the appropriate sections of its 
test report, but only in the “Operational Considerations” section of the test report. 


We incorporated the Deputy’s comments on the waiver process into the 
appropriate section of Finding B of the report. 


Management Comments Addressing Appendix FE, “Critical 
Operational Issues not Resolved Because of Approved Test 
Waivers or Test Limitations,” and Audit Response 


The Deputy commented on the F/A-18E/F operational effectiveness COIs, 
F/A-18E/F operational suitability COIs, reasons for waivers of test requirements, 
V-22 operational effectiveness COIs, and V-22 operational suitability COIs. 


F/A-18E/F Operational Effectiveness COIs. The Deputy stated that the report 
incorrectly states that, “As a result of the waivers, the COIs for operational 

effectiveness were not fully resolved during initial operational testing.” Further, 
he stated that, because of test waivers, COMOPTEVFOR could not fully resolve 
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the COI for air-to-ground sensors, which was | of the 20 operational effectiveness 
COIs. The Deputy stated that COMOPTEVFOR fully resolved the remaining 
19 operational effectiveness COIs. 


Audit Response. The COMOPTEVFOR test report did state that it was able to 
fully resolve 19 of the 20 operational effectiveness COIs; however, 
COMOPTEVFOR resolved those COIs without including an assessment of the 
effects of waivers on the resolution of the 19 COIs. This condition occurred 
because the Navy policy precludes COMOPTEVFOR from assessing the effect of 
waived requirements in its analysis on the resolution of COIs. In the audit report, 
the analysis considered the effect of the test waivers and deferrals on the 
resolution of the COIs. 


F/A-18E/F Operational Suitability COIs. 


Audit Response. 


Reasons for Waivers of Test Requirements. The Deputy disagreed with the 
following statement in the report: “Because of performance problems identified 
during developmental test, the Director of Navy Test and Evaluation and 
Technology Requirements, Office of the Chief of Naval Operations approved the 
following developmental test waivers.” He stated that the Director of Navy Test 
and Evaluation and Technology Requirements granted the following test waivers 
because the capability had not been tested before operational evaluation and not 
because of performance problems identified during developmental test: 


e Maximum short take off (shipboard) 48,500 pounds, rather than 
mission profile specific weight. 


e Aircraft not cleared for air combat maneuvering. 
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Further, the Director of Navy Test and Evaluation and Technology Requirements 
granted the following test waivers because the capability was still under 
development and had not been fielded before operational test and evaluation. 

e Defensive weapon system not available for test. 

e Crashworthy auxiliary fuel tanks not available for test. 

e Aircraft not cleared to operate in icing conditions. 

e Ground collision avoidance and warning system not available for test. 


Audit Response. We revised the report accordingly. 


V-22 Operational Effectiveness COIs. 


Audit Response. 


V-22 Operational Suitability COIs. 


Audit Response. 
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Appendix I. Audit Response to Air Force 
Comments on the Report 


The detailed responses to the comments from the Director, Air Force Test and 
Evaluation on statements in the draft report follow. The complete text of those 
comments is in the Management Comments section of this report. 


Management Comments on the Overall Report and Audit 
Response 


The Director commented on the DOT&E Memorandum; test waivers, limitations, 
and deferrals; evolutionary acquisition; fielding new capabilities; and input from 
the Military Departments. 


DOT&E Memorandum. The Director stated that the August 3, 2001, DOT&E 
memorandum referenced in the report did not reach Headquarters, Air Force Test 
and Evaluation or the Air Force Operational Test and Evaluation Center. Further, 
he stated that the Air Force Operational Test and Evaluation Center received a 
different memorandum, dated August 7, 2001; however, that memorandum did not 
contain any guidance about test and evaluation waivers. The Director stated that 
his office had since obtained a copy of the August 3, 2001, memorandum that 
DOT&E addressed to his staff. Had the Air Force been given the opportunity to 
review or coordinate on the August 3, 2001, memorandum, he stated that his office 
would have nonconcurred with the language about waivers and requirements. 
Further, he stated that his office was unsure whether the DOT&E disseminated his 
August 3, 2001, memorandum to the entire test and evaluation community. 


Audit Response. In the DOT&E memorandum, the DOT&E stated that he was 
communicating the major points of the memorandum to the Commanders of the 
Operational Test Agencies and was encouraging them to share the guidance with 
their staffs so that everyone involved would be on the same wavelength. Further, 
DOT&E stated that the Military Departments should already have promulgated the 
memorandum’s test policy. 


Even if the Director had been able to nonconcur with the DOT&E memorandum, 
he is required by statute to comply with DOT&E guidance. DOT&E has the 
authority to issue guidance to the Military Departments on operational test policy. 
Section 139, title 10, United States Code states that DOT&E is the principal 
adviser to the Secretary of Defense and the Under Secretary of Defense for 
Acquisition, Technology, and Logistics on operational test and evaluation in the 
DoD and is the principal operational test and evaluation official within the DoD 
senior management. Section 139 further states that DOT&E will prescribe, by 
authority of the Secretary of Defense, policies and procedures for the conduct of 
operational test and evaluation in the DoD. 


Test Waivers, Limitations, and Deferrals. The Director stated that the report 
identifies differences in opinion, policy, and practice among the Military 
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Departments and DOT&E staff about what constitutes a test waiver. The Director 
believes that the report incorrectly considers test waivers, test limitations, and test 
deferrals as the same. He stated that the Air Force rarely applies directly for test 
waivers, but, instead, identifies test limitations and test deferrals to DOT&E 
through the test plan approval process. Further, the Director stated that, after 
DOT&E reviews and approves the test plans, no additional action is required. He 
also stated that his office follows DoD 5000 policy and Title 10 that do not require 
waivers for test limitations or test deferrals. The Director stated that, nonetheless, 
his office is revising all Air Force test and evaluation policy documents and plans 
to address the use of waivers in more detail. In addition, he stated that the other 
Military Departments appear to use different strategies and policies for test 
waivers. 


Audit Response. Because the concept of the test waiver varies significantly 
among the Military Departments, we included in the Background section of the 
report the various definitions of test waiver used by DOT&E and the Military 
Departments. Because of the inconsistent procedures that the Military 
Departments use for test waivers and deferrals, the report recommends that the 
Under Secretary of Defense for Acquisition, Technology, and Logistics; the 
Director, Operational Test and Evaluation; the Deputy Under Secretary of the 
Army (Operations Research); the Assistant Secretary of the Navy (Research, 
Development, and Acquisition); and the Director, Air Force Test and Evaluation 
collectively establish consistent guidance for processing test waivers or deferrals. 


As stated in the report, DOT&E considers test limitations and deferrals as the same 
as test waivers although the Military Departments did not agree that test 
limitations and test deferrals are equivalent to test waivers. As defined by DOT&E, 
test limitations and test deferrals accomplish the same objective as test waivers by 
deferring required testing until later in the system acquisition process, usually after 
the system’s full-rate production decision. 


In reference to Air Force test waiver policy, Air Force Instruction 99-102 requires 
the system program manager to prepare a request to waive or modify fundamental 
provisions for operational test and evaluation. With user and developer 
concurrence, the system program manager is to submit the request through the 

Air Force Operational Test and Evaluation Center to the Air Force Test and 
Evaluation Directorate. The system program manager and the system’s program 
element monitor are to document any approved operational test and evaluation 
waivers in the TEMP and the program management directive, respectively. Further, 
Air Force Manual 63-119 requires a waived requirement to be tested in subsequent 
operational test and evaluations or the user must change the ORD requirement. As 
stated in the Air Force comments, the Air Force rarely adheres to those 
requirements for processing test deferrals to operational test requirements. 


We agree that DoD 5000 policy and Title 10 do not require the processing of 
waivers for test limitations or test deferrals. For tests requirements not resolved 
before the full-rate production decision, DoD Instruction 5000.2 requires that 
deficiencies encountered in testing before the LRIP decision be resolved before the 
system can proceed beyond LRIP and that any fixes to those deficiencies be 
verified during follow-on operational test and evaluation. 
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Evolutionary Acquisition. The Director stated that the report did not recognize 
that the evolutionary acquisition strategy in the latest DoD 5000-series is the 
preferred acquisition strategy. Further, he stated that the report did not recognize 
the principles of time-phased requirements discussed in Chairman of the Joint 
Chiefs of Staff Instruction 3170.01B, “Requirements Generation System,” 

April 15, 2001. The Director stated that his main point is that deferrals of some 
requirements and capabilities and the testing of the associated COIs and 
requirements must now be planned for so that the evolutionary acquisition strategy 
and the time-phased requirements will work. In addition, he stated that the 
materiel developers, testers, and users can no longer expect a technically perfect 
solution or “the perfect test” in the first increment that is deployed. In this regard, 
the Director stated that the Under Secretary of Defense for Acquisition, 
Technology, and Logistics mandated that trade-offs be made among cost, schedule, 
and performance so that development programs get interim capabilities delivered 
sooner while keeping cost and schedule under control. 


Audit Response. The key point of this report is to not just field new capabilities, 
but to field new capabilities that will fully meet warfighter requirements by 
efficiently resolving core impediments to fulfilling those requirements. Even if a 
system is being acquired using the evolutionary acquisition strategy, the system 
still has to meet the operational requirements and COIs for that block of the 
evolutionary acquisition. DoD Instruction 5000.2 requires that deficiencies 
encountered in testing before the LRIP decision be resolved before the system can 
proceed beyond LRIP, and that any fixes to those deficiencies be verified during 
follow-on operational test and evaluation. Therefore, deficiencies identified before 
each block of a system enters initial operational test and evaluation should be 
corrected before the system enters full-rate production for that block. Further, the 
objective of the evolutionary acquisition strategy is not to deliver a substandard 
system to the warfighter, but, as DoD Instruction 5000.2 states, the objective is to 
balance needs and available capability with resources, and to put capability into 
the hands of the user quickly through a block upgrade approach. 


Fielding New Capabilities. The Director stated that the three Air Force programs 
cited in the report, the Predator, the Joint Direct Attack Munition, and the F-15 
Tactical Electronic Warfare System, all used the evolutionary acquisition strategy 
and time-phased requirements to varying degrees to field new capabilities earlier 
than the traditional acquisition process. Further, he stated that the examples of test 
waivers, limitations, and deferrals cited in the report do not rise to a level of 
significance that warrant delaying any of these programs. The Director also stated 
that program managers are following the DoD 5000 guidance to field capabilities 
to warfighters sooner. 


Audit Response. The Air Force, by proceeding into full-rate production before 
resolving test deficiencies that were identified before the start of the initial 
operational test and evaluation, was not following DoD 5000 guidance. 


Input From the Military Departments. The Director stated that information in 
the draft appears to come solely from DOT&E with minimal to no input from the 
Military Departments. Further, he stated that no in-brief or interviews were held 
with Air Force testers. 
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Audit Response. We obtained information for the report from interviews with 
DOT&E; Army, Navy, and Air Force test and evaluation offices; and selected 
program offices. Specifically, within the Air Force, we met with representatives of 
the Air Force Test and Evaluation Directorate and the Air Force Test and 
Evaluation Center on June 8, 2001, and September 10, 2001, respectively. We 

also interviewed and obtained documentation from system program office 
personnel for the F-15 Tactical Electronic Warfare System, the Predator 
Unmanned Aerial Vehicle, and the Joint Direct Attack Munition Program Offices. 
For the Fighter Data Link, we obtained e-mail documentation from representatives 
of the Multifunction Information Distribution System Program Office. 


Management Comments Addressing the Main Body of the 
Report and Audit Response 


The Director commented on DoD waiver guidance, developmental test waivers, 
waiver equivalents, resolution of COIs, test responsibilities, missing data, DOT&E 
guidance, the F-15 Tactical Electronic Warfare System, the Joint Direct Attack 
Munition, definition of a test waiver, and Appendix A, “Scope and Methodology.” 


DoD Waiver Guidance. The Director stated that he agreed with the statement 
that “DoD guidance does not specifically define what a waiver from 
developmental and operational testing is.” He also stated that no officially 
coordinated definition or policy exists. The Director further stated that the only 
DoD-level test and evaluation waivers he knew of were in sections 2399 and 2366, 
title 10, United States Code, for system contractor involvement in operational test 
and evaluation and when live fire test and evaluation is too expensive and 
impractical. He stated that the Air Force recognizes and uses these kinds of 
waivers when necessary. However, he disagreed with the DOT&E definition of a 
waiver as a “deferral of a testing requirement, which does not eliminate the 
requirement for testing.” The Director stated that deferring the test of a system 
requirement to a later point in program development does not mean the need to test 
the requirement is waived. He also stated that the DOT&E definition of a waiver 
is not officially recognized DoD-wide, nor was it coordinated with the Military 
Department testers. The Director agreed that it is evident the concept of a test 
waiver varies greatly among the Military Departments. He stated that in addition 
to the Title 10 waivers, Air Force Instruction 99-102, “Operational Test and 
Evaluation,” July 1, 1998, requires “waivers” in the following three other 
instances: 


e Program managers must obtain a waiver from the Air Force Materiel 
Command to use non-DoD test facilities when such facilities are 
available; 


e The Air Force Operational Test and Evaluation Center must obtain a 
waiver from Headquarters, Air Force Test and Evaluation Directorate 
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before transferring responsibility for an operational test and evaluation 
program to a major command; and 


e The Air Force Operational Test and Evaluation Center must obtain a 
waiver from Headquarters, Air Force Test and Evaluation Directorate 
before altering, excluding areas from, or not conducting operational test 
and evaluation. The Air Force Operational Test and Evaluation Center 
needs user and developer concurrence before Headquarters, Air Force 
Test and Evaluation Directorate can process these kinds of waivers. 


Further, the Director stated that Air Force Manual 63-119, “Certification of 
System Readiness for Dedicated Operational Test and Evaluation,” February 22, 
1995, states that, “Approval of a waiver does not eliminate or alter the requirement 
for operational test and evaluation.” He also stated that the waived items must be 
tested in subsequent operational test and evaluation or the ORD must be changed. 


Audit Response. Because the concept of the test waiver varies significantly 
among the Military Departments, we included in the Background section of the 
report the various definitions of test waiver by DOT&E and the Military 
Departments. We also stated that the DOT&E definition does not include 
sections 2399 and 2366, title 10, United States Code, regarding system contractor 
involvement in operational test and evaluation and when live-fire test and 
evaluation is too expensive or impractical, respectively. Further, we stated that, 
although the Military Departments did not agree that test limitations and test 
deferrals were equivalent to test waivers, as defined by DOT&E,test limitations 
and test deferrals accomplished the same objective as test waivers by deferring 
required testing until later in the system acquisition process, usually after the 
system’s full-rate production decision. 


Because of the inconsistent procedures the Military Departments used for test 
waivers or deferrals, we recommended that the Under Secretary of Defense for 
Acquisition, Technology, and Logistics; the Director, Operational Test and 
Evaluation; the Deputy Under Secretary of the Army (Operations Research); the 
Assistant Secretary of the Navy (Research, Development, and Acquisition); and 
the Commander, Air Force Test and Evaluation Directorate collectively establish 
consistent guidance for processing test waivers or deferrals. 


In its discussion of approvals of waivers, Air Force Manual 63-119 states that 
“Waived items must be tested in subsequent operational test and evaluation or the 
ORD must be changed.” This statement supports the DOT&E assertion that test 
waivers accomplish the same objective as test deferrals by deferring required 
testing until later in the system acquisition process. 


Developmental Test Waivers. The Director stated that he did not know of any 
requirement for the Military Departments to obtain waivers for developmental test 
and evaluation of any kind. He further stated that DOT&E is restricted by 

section 139, title 10, United States Code, as follows: “The Director may not be 
assigned any responsibility for developmental test and evaluation, other than the 
provision of advice to officials responsible for such testing.” 
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Audit Response. Although the audit team’s review of the regulations did not 
indicate a requirement to obtain waivers for developmental test and evaluation, 
Air Force Instruction 99-101 requires that any approved waivers to developmental 
testing procedures or requirements be documented in the TEMP. We neither 
indicated in the draft report a requirement for the Military Departments to obtain 
waivers from developmental test and evaluation nor did DOT&E indicate that it 
had responsibility for developmental test and evaluation. 


Waiver Equivalents. The Director stated that the report alleges the existence of 
“waiver equivalents” called “test limitations” and “[test] deferrals.” Further, he 
stated that those waiver equivalents are not the same as a waiver. The Director 
stated that the report’s language generates confusion about what a waiver is and 
when it is required. He stated that the Air Force rarely applies outright for test 
waivers, but instead identifies test limitations and test deferrals to DOT&E through 
the test plan approval process. In addition, the Director stated that, once DOT&E 
reviews and approves the test plans to include the TEMP, additional action is not 
required. He stated that the Air Force follows the DoD 5000 series and Title 10, 
which do not require waivers for test limitations or test deferrals. Further, the 
Director stated that submission of a separate waiver request in addition to an 
approved test plan is a redundant action. He also stated that test_limitations are not 
test waivers, but are constraints to an ideal test strategy or plan imposed by 
circumstances largely beyond everyone’s control. The Director stated that test 
limitations are documented in test plans and TEMPs, along with ways to mitigate 
the impacts and still produce acceptable evaluations. In addition, he stated that test 
deferrals are not test waivers, but are agreements that certain parts of a test and 
evaluation program may be delayed until a later point in system development. The 
Director also stated that test deferrals are based on trade-off decisions mutually 
acceptable to developers, testers, and users, and are documented in test plans and 
TEMPs. 


Audit Response. To establish the definition of waivers, we contacted DOT&E to 
determine what the Director meant when he expressed concern about programs 
beginning operational testing without completing sufficient developmental test and 
evaluation and about the Navy’s use of test waivers. DOT&E defined a waiver as 
a deferral of a testing requirement, which does not eliminate the requirements from 
testing. In addition, Air Force Manual 63-119 states that “Waived items must be 
tested in subsequent operational test and evaluation or the ORD must be changed.” 
Therefore, a waiver defers testing to future periods. Test deferrals and limitations 
caused by inadequate system capabilities also defer testing to future periods. 
Further, in the Director’s comment, he stated that instead of applying for waivers, 
the Air Force identifies test limitations and test deferrals. Accordingly, the 

Air Force is identifying test limitations and test deferrals in TEMPs instead of 
processing test waivers to accomplish the same objective; that is, deferring 
required testing until later in the system acquisition process. 


Resolution of COIs. The Director stated that no requirement exists to resolve all 
COIs before proceeding into full-rate production or deployment for 
software-intensive systems. He stated that COIs are tester-developed 
requirements, not operational requirements, and that some COIs such as 
operational suitability require a much greater amount of testing so that the 
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operational requirement may not be satisfied until later. The Director further 
stated that the reformed acquisition environment allows the milestone decision 
authority to use trade-off analysis to defer the closure of COIs when it makes good 
sense for the program. 


Audit Response. Although deficiencies affecting the resolution of COIs do not 
need to be fixed and verified before the full-rate production decision, DoD 
Instruction 5000.2 requires that deficiencies encountered in testing before the 
LRIP decision be resolved before the system proceeds beyond LRIP at the full-rate 
production decision review, and that any fixes be verified during follow-on 
operational test and evaluation. In reference to tester developed COIs, we 
determined that testers use the ORD in developing the COIs. While all ORD 
requirements are not COIs, a correlation exists between the COIs that the testers 
develop and the operational requirements. To implement the testing requirement 
in DoD Instruction 5000.2, Air Force Operational Test and Evaluation Center 
Manual 99-102 states that, if every COI is resolved favorably, the system should 
be operationally effective and operationally suitable when employed in its intended 
environment by typical users. Air Force Instruction 99-102 states that initial 
operational test and evaluation will be planned to completely and unambiguously 
answer all COIs as thoroughly as possible, and will not defer testing into follow-on 
operational test and evaluation unless unavoidable. Air Force Instruction 99-102 
also states that follow-on operational test and evaluation will not be intentionally 
used as a backup for incomplete or poorly planned initial operational test and 
evaluation. 


Test Responsibilities. The Director stated that the fact that some COIs are not 
resolved before full-rate production or deployment is not caused solely by the 
testers, but is the final responsibility of the materiel developer and the milestone 
decision authority. He stated that the tester’s job is to provide information to the 
developer and the milestone decision authority within the constraints of the 
acquisition and the test and evaluation strategies. The Director also stated that, 
based on the COIs that were resolved and not resolved, the milestone decision 
authority makes the final decision to proceed into production, and not the testers. 


Audit Response. We agree, as stated in the report, that the testers’ inability to 
resolve COIs during initial operational test and evaluation was caused, to a large 
extent, by the Military Departments approving waivers or waiver equivalents. The 
term Military Departments includes program managers, testers, milestone decision 
authorities, and users. We also agree that it is within the milestone decision 
authority’s prerogative to make the decision to proceed into production when all 
COls are not resolved. The milestone decision authority, however should be 
informed of the effects of unresolved COIs on system performance so that an 
objective decision can be made. 


Missing Data. The Director stated that Appendix C, “Test and Evaluation 
Policy,” was missing along with pages 15 through 27. Further, he stated that 
DOT&E is not qualified to speak for the Military Departments regarding their test 
and evaluation policies. The Director also stated that we should have come to the 
Military Department testers to complete this section of the report. 
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Audit Response. The Director’s comment addressed the informal staffing of the 
discussion draft version of this report and not the draft report. 


DOT&E is qualified to speak for the Military Departments on their 
implementation of DoD test and evaluation policies. The DOT&E is empowered 
by statute and by authority of the Secretary of Defense to prescribe policies and 
procedures for the conduct of operational test and evaluation in the DoD. 


DOT&E Guidance. The Director expressed difficulty with the DOT&E 
memorandum, “DOT&E Goals,” August 3, 2001, and its attachment. He stated 
that DOT&E did not address the memorandum to the Military Departments, but 
only to the “DOT&E Staff,” and that DOT&E did not coordinate the memorandum 
with the Military Departments before publication. He stated that the memorandum 
included a few sentences about test waivers with which he fundamentally 
disagreed, such as, “There will be no waivers, or deferrals, to the completeness of 
evaluations. Requirements waived should be considered failed unless changed.” 
The Director stated that, if a testing requirement is deferred because a support 
asset was not available during test, this does not mean the system “failed” to meet 
the requirement. He further stated that those statements are too extreme and run 
counter to the spirit of streamlined acquisition in the current DoD 5000-series and 
the Chairman of the Joint Chiefs of Staff Instruction 3170.01B, “Requirements 
Generation System,” April 15, 2001. 


Audit Response. In the DOT&E memorandum, the DOT&E stated that he was in 
the process of communicating the major points of the memorandum to the 
Commanders of the Operational Test Agencies and was encouraging them to share 
the guidance with their staffs so that everyone involved would have the same 
understanding of the guidance. Concerning the testing deferral, if testing for a 
requirement is deferred because a support asset was not available during test, that 
requirement cannot be fully evaluated until the support asset is available; 
therefore, the completeness of evaluations is affected. Further, DoD 

Instruction 5000.2 requires that deficiencies encountered in testing before the 
LRIP decision be resolved before the system proceeds beyond LRIP at the full-rate 
production decision review, and that any fixes be verified during follow-on 
operational test and evaluation. 


F-15 Tactical Electronic Warfare System. Concerning the F-15 Tactical 
Electronic Warfare System, the Director stated that the report did not mention that 
DOT&E approved the Band 1.5 TEMP and operational test and evaluation plan, 
both of which addressed the false alarm rate and future testing. Further, he stated 
that the report did not mention that the Air Force submitted plans to DOT&E 
showing that it planned to remedy the false alarm rate problems in the next 
increment, which was the Band 5.0 increment. 


Audit Response. We revised the report as suggested. However, the point of the 
report is that COIs, which need to be demonstrated to prove that the system can 
perform its intended mission, were not resolved before the full-rate production 
decision, thereby the milestone decision authority allowed the system to continue 
into production creating the potential need for costly retrofit of fielded units and an 
initial operational performance that may be less than required to defeat the 
expected threat. 
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Joint Direct Attack Munition. The Director stated that the Joint Programmable 
Fuse System Program Office, not the Joint Direct Attack Munition System 
Program Office, is responsible for finding solutions to the interoperability problem 
identified in the report. Further, he stated that the follow-on operational test and 
evaluation is a normal followup to the initial operational test and evaluation 
regardless of whether the testers included the Joint Programmable Fuse in the 
initial operational test and evaluation. The Director also stated that the testers did 
not plan for the follow-on operational test and evaluation mainly because the Joint 
Programmable Fuse was not available. 


Audit Response. The Director addressed information contained in the discussion 
draft version of this report and not the draft report. Based on Air Force comments 
to the discussion draft, we determined that the Joint Direct Attack Munition did 
not have deficiencies that affected the resolution of COIs before the milestone 
decision authority made the full-rate production decision. 


Definition of a Test Waiver. The Director stated that the report contains the 
DOT&E definition of a test waiver. Further, he stated that the personnel from the 
Air Force Operational Test and Evaluation Center searched Title 10, past versions 
of the DoD 5000 series directives, the current OSD acquisition interim directives 
and guidance, and the current draft DoD 5000 series directives for the DOT&E 
definition of a test waiver. The Director also stated that his office cannot find the 
definition, policy, or guidance on waivers in those documents. In addition, he 
stated that personnel from the Air Force Operational Test and Evaluation Center 
recommend that, if DOT&E desires to establish a policy not allowing the waiver or 
deferral of testing operational requirements, it needs to document such policy in 
the current draft version of the DoD 5000 series directives. 


Audit Response. See the response to the “Test Waivers, Limitations, and 
Deferrals” issue on page 70. 


Appendix A, “Scope and Methodology.” The Director stated that the report does 
not identify the IG DoD team’s conclusion in determining the role of the Institute 
for Defense Analyses in the test and evaluation waiver process. Further, he stated 
that personnel at the Air Force Operational Test and Evaluation Center believe that 
the Institute for Defense Analyses has no role in the process and that it is a support 
contractor to DOT&E. 


Audit Response. In the “Scope and Methodology” section of the report, we stated 
that during the audit we determined the role of the Institute for Defense Analysis in 
the test and evaluation waiver process. As the Director commented, the Institute 
for Defense Analyses does not have a role in the test and evaluation waiver process 
other than as a support contractor to DOT&E. 
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Management Comments on Finding A and Audit Response 


The Director commented on management flexibility, resolution of COIs during 
initial operational test and evaluation, table of COIs not fully resolved, 
evolutionary development, test waiver decisions, Predator operational 
effectiveness issues, F-15 Tactical Electronic Warfare System test plans, Joint 
Direct Attack Munition issues, and DoD policy. 


Management Flexibility. The Director stated that a key qualification overlooked 
in the report paragraph “Air Force Policy,” and the rest of the report, was the 
words, “unless unavoidable.” Further, he stated that those words allow for 
unforeseen circumstances in development programs and test and evaluation 
strategies that would do great harm to programs if no management flexibility were 
allowed. The Director stated that, in some instances, the deferral of a test 
requirement or the completion of a COI is better than severely penalizing the 
program with cost and schedule effects resulting from a less than perfect 
development or test and evaluation program. He also stated that this qualification 
allows program managers, testers, and users to make reasonable trade-offs in the 
best interests of warfighters, according to the DoD 5000 series’ new streamlined 
acquisition principles. 


Audit Response. See the response to the “Resolution of COIs” issue to the 

Air Force on page 75. We agree that the milestone decision authorities have the 
prerogative to allow weapon systems that were not fully ready for production to 
continue into production when COIs for weapon systems are not fully resolved 
during initial operational test and evaluation. However, this action creates a 
potential need for costly retrofit of fielded units later and an initial operational 
performance that may be less than is required to defeat the expected threat. 


Resolution of COIs During Initial Operational Test and Evaluation. The 
Director stated that, occasionally, some COIs in initial operational test and 
evaluation cannot be resolved before full-rate production or deployment because of 
the ORD requirement or test limitations, or both. Further, he stated that a 
suitability requirement for a 1,000-hour mean time between failure for an aircraft 
system may require up to 5,000 flying hours in some instances. The Director 
stated that to require a system to log 5,000 flying hours before making a full-rate 
production or deployment decision may not be possible or good program 
management. He also stated that other tools, such as reliability growth curves, can 
reduce risk and help make good projections when those kinds of test limitations 
occur. The Director reiterated his statement that the draft report confuses test 
waivers with test limitations and test deferrals. 


Audit Response. Although COIs may not be fully demonstrated before the 
full-rate production, DoD Instruction 5000.2 requires that deficiencies 
encountered in testing before the LRIP decision be resolved before proceeding 
into full-rate production and any fixes verified in follow-on test and evaluation. In 
the report, we addressed deficiencies (test waivers) that affected the resolution of 
COIs and that were encountered before initial operational test and evaluation that 
were not resolved before the milestone decision authority made the full-rate 
production decision. Deficiencies encountered during initial operational test and 
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evaluation are considered an acceptable part of the initial operational test and 
evaluation process and, therefore, were not addressed in this report. In reference 
to the definitions of test waivers, test limitations, and test deferrals, we explained 
the rational used by DOT&E for considering test limitations and deferrals as the 
equivalent to test waivers in the Background section of the report. 


Table of COIs Not Fully Resolved. The Director stated that the table, “COIs Not 
Fully Resolved Because of Approved Test Waivers of Test Limitations for the 
Eight Defense Systems Affected,” was not balanced because it focused almost 
solely on suitability COIs. He reiterated his comments on the resolution of COIs 
during initial operational test and evaluation. The Director also stated that 
evolutionary acquisition strategies recognize that test limitations for COIs may 
require deferral of a final answer to resolve the requirement or the COI until a later 
date. 


Audit Response. The table addresses deficiencies that affected the resolution of 
COIs that were encountered before initial operational test and evaluation but were 
not resolved before the milestone decision authority made the full-rate production 
decision. With regard to the suitability COIs, the predominate COIs not resolved 
for the eight programs reviewed were suitability issues. Concerning evolutionary 
acquisition, we disagree that evolutionary acquisition strategies recognize that 
COIs for each block of the acquisition may require deferral of a final answer to 
resolve the COI until a later date. The operational testers are to establish COIs 
only for each block for performance capabilities that need to be demonstrated 
during that block development. 


Evolutionary Development. The Director stated that the draft report states that 
the Joint Requirements Oversight Council deferred some test requirements for the 
Patriot Advance Capability-3 program as part of an evolutionary development 
program. He stated that this deferral was a good example of how evolutionary 
acquisition works to ensure earlier delivery of capabilities to warfighters. Further, 
the Deputy stated that evolutionary acquisition is the “preferred acquisition 
strategy” in DoD. He also stated that we were not justified in criticizing the 
Patriot Advance Capability-3 program for deferring a requirement after careful 
deliberation with the JROC and deploying much-needed capabilities to protect our 
warfighters. 


Audit Response. The draft report does not debate the Joint Requirements 
Oversight Council process or how fast capabilities are delivered to the warfighters. 
The point of the report is that COIs, which are required for a system to perform its 
mission, were not resolved before full-rate production, thereby allowing the system 
to continue into production and creating the potential need for costly retrofit of 
fielded units and an initial operational performance that may be less than required 
to defeat the expected threat. As discussed above, the Patriot Advanced 
Capability-3 program is not a good example of how an evolutionary acquisition 
works because COIs that were established for a specific block of the acquisition 
strategy were not resolved before the production decision. 


Test Waiver Decisions. The Director stated that the report’s discussion of the 


nine COI categories affected by test waivers and limitations and examples of the 
affected COIs from the eight programs affected focused on relatively minor 
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problems that did not warrant delaying the full-rate production or deployment 
decisions. Further, he stated that perfect development programs and test and 
evaluation programs did not exist and that waiting for every problem to be 
resolved, no matter how small, before full-rate production or deployment would 
result in no systems being deployed, or at best would add totally unacceptable cost 
and delay to programs. The Director also stated that the decision to waive test 
requirements or defer some tests is a mutual agreement among the testers, materiel 
developers, and users based on program and technical risk weighed against the risk 
of not deploying new capabilities to our warfighters. He stated that sometimes 
“better” is the worst enemy of “good enough.” 


Audit Response. The Office of the DOT&E provided the list of programs that 
were issued waivers that we used for the audit. As stated in the report, we 
conducted the audit based on the DOT&E “Operational Test and Evaluation 
Annual Report, FY 2000,” in which the Director expressed concern about 
programs beginning operational testing without completing sufficient 
developmental test and evaluation and about the Navy’s use of test waivers. The 
Defense Science Board also expressed similar concerns in the “Defense Science 
Board Task Force on Test and Evaluation Capabilities,” December 2000. Systems 
provided to the warfighter should be operationally effective and suitable. The 
report addresses test waivers that affected the resolution of COIs before the initial 
operational test and evaluation and that were not resolved before the milestone 
decision authority made the full-rate production decision. When a system enters 
initial operational test and evaluation with known deficiencies, those deficiencies 
should be resolved before entering full-rate production. If those deficiencies are 
not resolved before full-rate production, especially deficiencies effecting COIs 
required for a system to perform its mission, the system continues into production 
and thereby creates the potential need for costly retrofit of fielded units, and initial 
operational performance from the system that may be less than required to defeat 
the expected threat. 


Predator Operational Effectiveness Issues. The Director stated that the report 
criticizes the Predator Unmanned Aerial Vehicle program for not achieving its full 
potential before full-rate production and deployment. Further, he stated that the 
program was intended as a fast-track program from the start and follows the 
evolutionary acquisition model in the new DoD 5000 series. He also stated that 
we were not justified in criticizing the Predator solely on technical and testing 
grounds because the Predator delivered new capabilities to warfighters earlier just 
as the new DoD 5000-series envisions. Further, the Director stated that DOT&E 
criticism of the Predator has little credibility because top-level combatant 
commanders had the highest praises for the Predator despite its drawbacks. The 
Director noted that latest draft DOT&E Annual Report to Congress withdrew last 
year’s conclusion that Predator was not effective and not suitable. 


Audit Response. The fact remains that the Predator did not meet its COI for 
effectiveness before full-rate production. The Predator experienced problems 
recognizing tactical-sized targets, as required as a critical mission function, and 
has continued into production and created the potential need for costly retrofit of 
fielded units to meet its critical mission requirements. However, if recognizing 
tactical-sized targets is not a COI, the cost for this additional requirement should 
be avoided. 
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F-15 Tactical Electronic Warfare System Test Plans. The Director stated that 
the report criticizes the test plans for the F-15 Tactical Electronic Warfare System 
for not showing test limitations for an excessive built-in-test false alarm rate. 
Further, he stated that DOT&E knew about the deficiency; therefore, no waivers 
were requested or submitted. The Director also stated that DOT&E approved the 
test plans as submitted without commenting on the lack of test limitations. In 
addition, he stated that test plan approval covered the test limitations and test 
deferrals. Further, the Director stated that DOT&E knew that the Air Force would 
attempt to correct the deficiency in the next incremental release, which is how 
evolutionary acquisition is supposed to work. 


Audit Response. We agree that DOT&E was aware of the excessive built-in-test 
false alarm rate, as stated in the report. However, this awareness does not alleviate 
the requirement for the system to meet the operational requirements and COIs for 
that block of the evolutionary acquisition. Because the deficiency was identified 
before that the system entered initial operational test and evaluation, the program 
office should have resolved the deficiency before the system entered full-rate 
production for that block. The objective of the evolutionary acquisition strategy is 
not to deliver a substandard system to the warfighter, but, as DoD 

Instruction 5000.2 states, the objective is to balance needs and available capability 
with resources, and to put capability into the hands of the user quickly. 


Joint Direct Attack Munition Issues. The Director had multiple comments 
concerning the Joint Direct Attack Munition. Specifically, 


e Headquarters, Air Force Test and Evaluation did not require approval 
of waivers or test deferrals of any kind for this program. While the 
Air Force recognized that some tests were deferred, Headquarters, 

Air Force Test and Evaluation kept close watch on those matters when 
reviewing operational test and evaluation plans before submission to 
DOT&E. 


e The Air Force Operational Test and Evaluation Center decertified the 
Joint Programmable Fuse for operational test and evaluation, not the 
Joint Direct Attack Munition. This test limitation was outside the 
control of the Joint Direct Attack Munition Program Manager and 
should not have held the Joint Direct Attack Munition program hostage. 
Other fuses were available that afforded adequate capabilities to 
demonstrate the Joint Direct Attack Munition. Fuse delays were only a 
small part of the total weapons effects equation, so the total effect on 
warfighters of not having the Joint Programmable Fuse was minimal. 


e The Joint Direct Attack Munition re-targeting capabilities were not 
affected by lack of Joint Programmable Fuse. Re-targeting was a 
capability inherent in the Joint Direct Attack Munition and not 
dependant on the fuse. The only problem is if the Joint Direct Attack 
Munition were retargeted in flight, the fuse setting may not always be 
precisely compatible with the new target. For troops in contact, who 
need ordnance right away, this criticism does not pass the “so what” 
test. 
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Audit Response. The Director addressed information contained in the discussion 
draft version of this report and not the draft report. Based on Air Force comments 
to the discussion draft, we determined that the Joint Direct Attack Munition did 
not have deficiencies that affected the resolution of COIs before the milestone 
decision authority made the full-rate production decision. 


DoD Policy. The Director states that the report cites the DOT&E Memorandum, 
August 3, 2001, as the DoD-level policy for directing the operational test agencies 
in the Military Departments to not use waivers or deferrals that prevent a complete 
evaluation of operational requirements. Further, he stated that personnel at the 
Air Force Operational Test and Evaluation Center do not consider this memo as 
policy, but rather guidance to DOT&E staff to implement the goal and objective of 
“rigorous and robust testing that is adequate.” The Deputy also stated that this 
same guidance was provided as a courtesy to the Commander, Air Force 
Operational Test and Evaluation Center in a subsequent letter, August 7, 2001. In 
addition, he stated that DOT&E policy is officially promulgated through DOT&E 
policy letters and that the Air Force Operational Test and Evaluation Center 
immediately implements these policy letters upon receipt through updates to 

Air Force Operational Test and Evaluation Center directives. 


Audit Response. See the response to the “DOT&E Memorandum” issue on 
page 69. 


Management Comments on Finding B and Audit Response 


The Director commented on DOT&E guidance; Air Force test policy; test 
limitations in test plans; revision of Air Force instructions; Air Force 

Instruction 99-102; testing of waived test requirements; source of information; 
waivers for developmental testing; inconsistent waiver policy; changes to DoD 
draft policy; Air Force nonconcurrence with DOT&E policy; waiver policies and 
procedures; and Air Force procedures, “Test and Evaluation Waivers.” 


DOT&E Guidance. The Director stated that DOT&E did not issue its August 3, 
2001, guidance to the Air Force, and the Air Force was not afforded a chance to 
coordinate on the guidance. 


Audit Response. See the response to the “DOT&E Guidance” issue on page 76. 


Air Force Test Policy. The Director did not agree that “Air Force procedures 
require the processing of waivers to defer specific operational test requirements to 
the future.” He stated that, while Air Force test policies do address waivers, those 
policies are not so specific. The Director stated that the Air Force addresses test 
limitations and test deferrals through the test plan approval process. He also stated 
that, during this process, the Air Force’s intention is that Air Force 

Manual 63-119, paragraph 4.2.7, be followed: “Approval of a waiver does not 
eliminate or alter the requirement for OT&E [operational test and evaluation]. 
Waived items must be tested in subsequent OT&E or the ORD must be changed.” 
In addition, the Director stated that the Air Force accomplishes the same goal in 
the end. 
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Audit Response. The Director’s comment addressed information in the discussion 
draft version of this report and not the draft report. The draft report correctly 
stated the Air Force waiver procedures. 


Test Limitations in Test Plans. The Director stated that the Air Force has no 
Headquarters, Air Force-level policies addressing procedures for the program 
managers to use test limitations in test plans to defer specific operational test 
requirements to the future. Further, he stated that he did not know of any major 
command policies in this area either. 


Audit Response. The Director’s comments addressed information in the 
discussion draft version of this report and not the draft report. In the draft report, 
we revised the report as stated in our response to the “Air Force Test Policy” issue 
on page 82. Concerning Air Force policy, Air Force Manual 63-119 requires that 
the description of operational test and evaluation test limitations that may affect 
the full-rate production decision be placed in the TEMP. The Air Force use of test 
limitations to defer testing has the same effect as a test waiver. 


Revision of Air Force Instructions. The Director stated that Air Force test and 
evaluation policies are under revision. A new Air Force Instruction 99-103, 
“Air Force Test and Evaluation,” is already in a mature stage of development and 
will address this area. 


Audit Response. In issuing Air Force Instruction 99-103, the Air Force needs to 
ensure that the new instruction is consistent and compliant with DOT&E guidance 
on test waivers and deferrals. 


Air Force Instruction 99-102. The Director stated that the Air Force has not 
published or is not aware of a January 20, 2000, version of Air Force 

Instruction 99-102. He asked whether this instruction could be Air Force 
Operational Test and Evaluation Center Instruction 99-102. The Deputy stated 
that his office did not approve Air Force Operational Test and Evaluation Center 
Instruction 99-102 before publication and did not review it in conjunction with his 
office’s review of the draft of this report. 


Audit Response. The Director’s comments addressed information in the 
discussion draft version of this report and not the draft report. The draft report 
cited Air Force Operational Test and Evaluation Center Instruction 99-102, 
“Operational Test and Evaluation,” January 20, 2000. 


Testing of Waived Test Requirements. The Director stated that Air Force 
Manual 63-119, not Air Force Instruction 99-102, requires Air Force program 
managers to “list any required waivers or areas excluded from OT&E [operational 
test and evaluation]” during the certification process and in the certification 
message. Further, he stated that the Manual states that “approval of a waiver does 
not eliminate or alter the requirement for OT&E. Waived items must be tested in 
subsequent OT&E or the ORD must be changed.” 


Audit Response. The Director’s comments addressed information in the 


discussion draft version of this report and not the draft report. The draft report 
referenced Air Force Manual 63-119 concerning testing of waiver requirements. 
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Source of Information. The Director stated that his staff could not locate the 
following statement in Air Force instructions: “identify test limitations in test plans 
to defer the demonstration of specific operational test requirements.” He asked 
whether this statement could be in Air Force Operational Test and Evaluation 
Center Instruction 99-102. 


Audit Response. The Director’s comments addressed information in the 
discussion draft version of this report and not the draft report. The draft report 
states that “The Air Force procedures require program managers to process 
waivers to defer specific operational test requirements and identify test limitations 
in test plans.” This statement paraphrases information in Air Force 

Manual 63-119. 


Waivers for Developmental Testing. The Director stated that, although 

Air Force Manual 63-119 uses the term “waiver,” Headquarters, Air Force Test 
and Evaluation does not require the Air Force Operational Test and Evaluation 
Center to submit formal paperwork for “acceptance” of an operational test and 
evaluation with test limitations or “waivers.” Further, he stated that his office does 
not require waivers for missing developmental test and evaluation areas. The 
Director stated that his office leaves that decision between the Air Force 
Operational Test and Evaluation Center commander and the program’s certifying 
official, who is usually the program executive officer. He also stated that the 

Air Force Operational Test and Evaluation Center commander makes the final 
decision, but must still have his test plan coordinated through Headquarters, 

Air Force Test and Evaluation and approved by DOT&E before starting the 
operational test and evaluation. 


Audit Response. The Director’s comments addressed information in the 
discussion draft version of this report and not the draft report. The draft report 
states that Air Force Manual 63-119 requires a waived requirement to be tested in 
subsequent operational test and evaluations or the user must change the ORD 
requirement. The draft report also states that the Manual requires the system 
program manager, before initial operational test and evaluation, to: 


e list any required waivers or areas excluded from the test and evaluation, 
the rationale, and future plans to clear the waivers, and 


e provide a summary of the list to the operational tester. 


In response, the operational tester is to indicate whether to proceed with the test 
and is to discuss the effects of test limitations and test waivers on resolving 
operational test issues in the test report. 


Inconsistent Waiver Policy. The Director stated that he agreed with the 
conclusion that the Military Departments do not have consistent policies, 
procedures, or terminology for test waivers and limitations. 


Changes to DoD Draft Policy. The Director stated that the report incorrectly 
states that the 120-day interim guidance requires that deficiencies encountered in 
testing before the LRIP decision be resolved before the system can proceed 
beyond LRIP, and that any fixes to those deficiencies be verified during initial 
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operational test and evaluation. He stated that the words “initial operational test 
and evaluation” should read “follow-on operational test and evaluation” as stated 
in the latest draft version of the DoD Instruction 5000.2, paragraph 3.7.6, which 
was pending signature. The Director stated that the change is very significant. 


Audit Response. The Director’s comments addressed information in the 
discussion draft version of this report and not the draft report. The draft report 
states that the 120-day interim guidance requires that any fixes to those 
deficiencies be verified during follow-on operational test and evaluation. 


Air Force Nonconcurrence with DOT&E Policy. The Director stated that the 
Air Force did not review or coordinate on the August 3, 2001, DOT&E 
memorandum and would have nonconcurred with the guidance on waivers and 
requirements. Further, he stated that he and his office do not know what guidance 
the other Military Departments received from DOT&E. The Director is unsure 
whether DOT&E disseminated the August 3, 2001, memorandum to the test and 
evaluation communities in the Military Departments. 


Audit Response. See the response to the “DOT&E Memorandum” issue on 
page 69. 


Waiver Policies and Procedures. The Director stated that the Air Force 
Operational Test and Evaluation Center did not have a policy to waive or defer 
operational test requirements. Further, he stated that the Commander, Air Force 
Operational Test and Evaluation Center supports the DOT&E goal of rigorous, 
robust, and adequate testing without waivers or deferrals by implementing the 
policy outlined in Air Force Operational Test and Evaluation Center 

Instruction 99-103. The Director also stated that the Commander, Air Force 
Operational Test and Evaluation Center will not accept certification of system 
readiness for operational test and evaluation until the developer can demonstrate 
stabilized performance under an operational (stressed) environment with a 
production representative article. In addition, the Director stated that test teams at 
the Air Force Operational Test and Evaluation Center are aware of requirements in 
Air Force Manual 63-119, “Certification of System Readiness for Dedicated 
Operational Test and Evaluation,” that are critical to ensuring, in collaboration 
with the applicable system program office, that the system can be certified ready 
for operational test and evaluation. 


Audit Response. See the response to the “DOT&E Memorandum” issue on 
page 69. 


Air Force Procedures, Test and Evaluation Waivers. The Director stated that 
the report refers to Air Force Instruction 99-102, “Operational Test and 
Evaluation,” when defining the Air Force’s policy on test and evaluation waivers. 
Further, he stated that the report quotes Air Force Instruction 99-102, 

paragraph 2.5, and that personnel at the Air Force Operational Test and 
Evaluation Center believe that the report incorrectly interprets that paragraph. 

The Director also stated that personnel at the Air Force Operational Test and 
Evaluation Center refer to this paragraph when submitting a waiver to the 
requirement for operational test and evaluation and not to waive or defer testing of 
operational requirements. In addition, he stated that the report incorrectly states 
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that the system program manager is the individual that submits waivers under 

Air Force Instruction 99-102, paragraph 2.5. Further, the Director stated that 
program management at the Air Force Operational Test and Evaluation Center 
submits the waiver request, with user and developer concurrence, to the Air Force 
Test and Evaluation Directorate. 


Audit Response. Air Force Instruction 99-102, paragraph 2.5, “Requests to 
Modify the Provisions for OT&E [Operational Test and Evaluation],” states that: 


Requests to modify (waive) the fundamental provisions of this AFI 
[Air Force Instruction] for any OT&E must be submitted in writing, 
with user and developer concurrence, through AFOTEC/CV [Air Force 
Operational Test and Evaluation Center/CV] to HQ USAF/TE 
[Headquarters, Air Force Test and Evaluation]. The SM _ [system 
manager] will document any approved OT&E “waivers” in the TEMP, 
and the system’s program element monitor (PEM) will document OT&E 
“waivers” in the PMD [program management directive]. 


Based on the Director’s comments, we revised the paragraph to state that the 

Air Force Operational Test and Evaluation Center instead of the system program 
manager prepares a request to waive or modify fundamental provisions for 
operational test and evaluation. With user and developer concurrence, the 

Air Force Operational Test and Evaluation Center submits the request to the 

Air Force Test and Evaluation Directorate. 
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Office of the Secretary of Defense 
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Department of the Army 
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Auditor General, Department of the Army 
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Commandant of the Marine Corps 

Aviation Department 
Assistant Secretary of the Navy (Manpower and Reserve Affairs) 
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Congressional Committees and Subcommittees, Chairman and 
Ranking Minority Member 


Senate Committee on Appropriations 

Senate Subcommittee on Defense, Committee on Appropriations 

Senate Committee on Armed Services 

Senate Committee on Governmental Affairs 

House Committee on Appropriations 

House Subcommittee on Defense, Committee on Appropriations 

House Committee on Armed Services 

House Committee on Government Reform 
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Director, Operational Test and Evaluation 
Comments 


OFFICE OF THE SECRETARY OF DEFENSE 
1700 DEFENSE PENTAGON 
WASHINGTON, DC 20301-1700 


SetRATIGNAL TEST April 24, 2003 


AND EVALUATION 


MEMORANDUM FOR INSPECTOR GENERAL, DEPARTMENT OF DEFENSE 


SUBJECT: Report on the Management of Developmental and Operational Test Waivers for 
Defense Systems (Project No. D2001 AE-0123) 


Thank you for the opportunity to review and provide comments concerning your draft 
report on the management of test waivers within the Department. I believe that your 
recommendations will be useful. 


We seek to limit the issuance of waivers and eliminate their intrusion into evaluations of 
operational effectiveness and suitability. In your report, you recommend that the Services update 
their policies, practices, and procedures for managing waivers. We concur. Progress has already 
been made in this area, and your administrative recommendations will help ensure proper 
management in that regard. 


Additional comments on the draft report are attached. 


Vosscier gate 


Thomas P. Christie 
Director 


Attachment: 
As stated 


% 
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Final Report 


Reference 


Revised 
Page 1 


Revised 
Page 48 


Comments on the DoDIG; Draft Report on the Management of Developmental and 
Operational Test Waivers for Defense Systems (Project No. D2001 AE-0123), 
February 20, 2003 


1. Page 1, Background, second paragraph: Change last sentence to read “...evaluation 
and when live-fire test and evaluation...” from “..,evaluation and when live-five test 
and evaluation...” Reason: Correciness. 


2. Report Recommendation A.2. Milestone decision authorities require that program 
managers resolve critical operational issues before approving full-rate production for 
a weapon system. 


¢ PARTIALLY CONCUR. There may be examples when, due to unanticipated 
problems (test range limitations, instrumentation limitations and/or availability, 
etc.), all COIs may not be fully evaluated at OT&E, while due to an operational 
requirement a capability is urgently needed. In such a case, it may be more 
beneficial to the warfighter to field that current capability rather than waiting to 
enter FRP until all COIs are resolved. Milestone Decision Authorities should 
have that prerogative. 


3. Clarification: Page 40, Fighter Data Link (FDL). During IOT&E, inadequate flight 
hours were available to evaluate FDL BIT and logistics supportability. While 
1,000 hours were required, during Follow-on Test and Evaluation, the FDL was able 
to demonstrate 963 hours. 
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Department of the Army Comments 


DEPARTMENT OF THE ARMY 
DEPUTY UNDER SECRETARY OF THE ARMY 
OPERATIONS RESEARCH 
102 ARMY PENTAGON 
WASHINGTON DC 20310-0102 


APR 1 8 2003 


MEMORANDUM FOR OFFICE OF THE INSPECTOR:GENERAL OF THE 
DEPARTMENT OF DEFENSE (MARY L. UGONE) 


SUBJECT: Draft Report —- “Management of Developmental and Operational Test 
Waivers for Defense Systems,” dated February 20, 2003, Project No. D2001AE-0123 


Per your request, | hereby provide the Army management comments to the two 
recommendations contained in the subject report: 


Recommendation A. Concur. HQDA-approved critical operational issues and 
criteria are focused on mission accomplishment. Per Army T&E policy, a breach of a 
specific criterion is reason to delay entry into full-rate production unléss other evidence 
of acceptable system operational effectiveness and suitability is provided. 


Recommendation B. Nonconcur. Referenced DOT&E memorandum was 
directed to the DOT&E staff and focused on the completeness of evaluations. Applying 
consistent test waiver and deferral guidance throughout the Department requires 
definition of terms. Propose this recommendation be modified to reflect the 
establishment of a DoD-wide process action team, chaired by the USD(AT&L) and 
DOT&E, with membership from the Military Department T&E headquarters and 
Operational Test Agencies. Thus, consistent test waiver and test deferral guidance 
would exist throughout the Department. 


Additional comments dealing with other areas of the report are enclosed. 


Lidl. Ut Pe~ 


Encl Walter W. Hollis 
as Deputy Under Secretary of the Army 
(Operations Research) 


Printed on CK) Recycled Paper 
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Final Report 
Reference 


Revised 
Page 19 


Page 5 


Additional Comments 
to 
Draft Report — “Management of Developmental and Operational Test Waivers 
for Defense Systems,” dated February 20, 2003, Project No. D2001AE-0123 


1. The stated genesis of this audit was the DOT&E concern about ‘programs 
beginning operational test and evaluation without completing sufficient 
developmental test and evaluation.’ Scope of this audit, limited to the test waiver and 
test deferral process, was not sufficiently broad enough to yield findings or 
recommendations that could improve the robustness of developmental testing or the 
likelihood of conducting a successful operational test. 


2. Recommend that the wording in the first paragraph, top of page 16 (OSD 
Guidance on Test Deferrals) be revised. Of concer is the sentence that reads - 
“Further, the DOT&E issued guidance on August 3, 2001, that directed Military 
Department testers not to issue waivers or deferrals that would prevent a complete 
evaluation of operational requirements.” Per Army T&E policy, testers are not 
authorized to waive documented test requirements. Modification of Recommendation 
B as proposed by the DUSA(OR), will enable the DOT&E guidance to be directed by 
the USD(AT&L) to the Service Acquisition Executives, not the testers. 


3. Appendix C. Test and Evaluation Policy. Recommend that the COI approval 
authority be mentioned for each of the Military Departments. Being key decision 
maker operational concerns, stating the approval authorities will reveal how the COI 
and COIC are being considered (that is, as a tester product or a decision-maker 
product). Per AR 73-1, the COIC approval authority is based upon the type of 
program — (1) Deputy Chief of Staff, G-8 (Programs) for materiel and tactical C4I/IT 
programs and (2) Army ClO/G-6 for non-tactical C4I/IT programs. Army T&E policy 
also addresses a mandatory attachment entitled ‘Requirements/Test Crosswalk 
Matrix.’ Accordingly, each source for each operational requirement is identified and 
agreed upon with the T&E Master Plan approval. 


4. Program Specific Comments 
a. Section 1 — JAVELIN. 


(1) Recommend removing the JAVELIN from the draft DoD IG report, as it does 
not support the overall argument of unresolved issues causing costly retrofits of 
fielded units or an initial operational performance that is less than expected to 
defeat the threat. 

(2) Given that JAVELIN remains in the draft DoD IG report, propose the following: 
(a) Addressing Critical Operational Issues, page 5, Table of Critical 
Operational Issues. Non-concur. Rationale: Of the nine COI reviewed, 
Javelin is incorrectly identified as failing in one, reliability. 


Enclosure 1 
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(b) Reliability, Availability, and Maintainability, page 8, Javelin paragraph and 
Operational Suitability Issues, page 31, Critical Operational Issue (Reliability). 
Non-concur. Rationale: The 100 hours requirement cited, is an old JSOR 
requirement that was written before the hardware configurations of the tactical 
and training systems were firm. The FTT (range) major components include a 
Command Launch Unit (CLU), a simulated round (SR) and an Instructors 
Station (IS). The U.S. Army Infantry School reevaluated the requirements 
prior to the AMSAA Independent Evaluation in 1994 and reduced the 100 
hours to 50 hours. This 50 hours requirement was established for System 
Maturity, which is MS IIl+3 years. The requirement at Milestone (MS) III, as 
established by reliability projections, is 32 as reported in the ATEC System 
Evaluation Report (SER) for the Javelin Anti-armor Weapon System, 27 March 
1997. The 25 hours still did not quite reach the 32 hours requirement (32 vs. 
50), but it is much closer than 25 vs. 100, and certainly would not have caused 
us to not field the Javelin weapon system. The majority of the problems with 
the FTT (range) were related to the CLU-to-SR connector and latch, battery 
cables, and battery. Most of the fixes were simple and effective. This was 
evidenced by the FTT (range) reliability estimate of 62 hours in the ATEC 
System Assessment of Javelin Reliability and Availability Maturity of 
December 2000. 

(c) Reliability, Availability, and Maintainability, page 8, Javelin paragraph and 
Operational Suitability Issues, page 31 Critical Operational Issue (Reliability), 
Chronology of events. Non-concur. Rationale: The last sentence in the 
Javelin paragraph incorrectly states that the Army operational testers resolved 
those suitability issues after the full-rate production (FRP) decision. ATEC 
conducted a follow on limited user test (LUT) in March through May 1996, 
primarily to resolve the FTT suitability issues. The FTT successfully met 
suitability criteria. Following the Javelin LUT and an additional Customer 
Confirmatory Test, again conducted by ATEC, the ASARC provided a MSIII 
FRP decision in July 1997. The FRP was not obtained until the required 
additional testing was successfully completed demonstrating the FTT 
Suitability. 

(d) Page 5: Add training category to table. 

(e) Page 28: Include the description of the FTT (range) in the description of 
Javelin as this is the only issue. 


b. Section Il - PATRIOT PAC-3. 


(1) Cancellation of requirements general comment: As of January 2002, all 
Missile Defense Systems Operational Requirements Documents (ORDs) have 
been cancelled retroactively by the SECDEF and all future developments would 
be capabilities based. 

(2) Operational Effectiveness/Issues, page 6 and 32, deferred requirements and 
test incident reports (TIRs). Non-concur. Rationale: Based on delays in software 
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Page 38 


Page 5 
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development and PAC-3 missile availability, a decision was made in November 
1998 to split the PATRIOT PAC-3 Configuration-3 system testing, evaluation and 
fielding into two parts-First Unit Equipped-Ground (FUE-G) and First Unit 
Equipped-Missile (FUE-M). The FUE-G Limited User Test (LUT) was conducted 
March through October 2000. Test Incident Reports (TIRs) noted during this test 
were included in the FUE-G Conditional Materiel Release (CMR), and were 
subsequently resolved or deferred prior to the FUE-M IOTE conducted January 
through May 2002. Some of the deferred TIRs will be resolved through post - 
IOTE events. Even though the ORD has been cancelled and some of the 
requirements have been deferred, they are still valid user requirements and will 
be tested and evaluated at a later time. 

(3) Operational Effectiveness/Issues, pages 6 and 31, PAC-3 paragraphs. Non- 
concur. The draft DOD IG report states “The testers were not able to resolve the 
operational effectiveness COI before the planned full rate production decision 
because the Army deferred test requirements.” It is correct that there were 
deferrals of test requirements driven by deferrals of system performance 
requirements. However, the testers believe this was a prudent decision (based 
on world events) with acceptable risks. Rationale: The Inspector General, DOD 
based this audit report on the “Operational Test and Evaluation Annual Report, 
FY 2000” by the Director, Operational Test and Evaluation (DOT&E) in February 
2001. Therefore, the comments related to PAC-3 system were based on the data 
obtained during Limited User Test (LUT) and Regression Test held in March 
through October 2000. This LUT was conducted to assess the improvement in 
the PAC-3 Ground Equipment operating with PAC-2/GEM missile only. A 
conditional materiel release was approved following the completion of the LUT 
and regression testing, in order to support the Ground Equipment LRIP. Deferred 
requirements from the TIRs were attached as the required conditions for materiel 
release. ATEC understood before the LUT that the PAC-3 missile system is 
required to meet the operational effectiveness COI, and a full-up system IOTE 
was conducted in January through May 2002 to assess the increased capability of 
the PATRIOT system integrated with PAC-3 missile. At the completion of the 
IOTE and regression testing, an urgent materiel release for the PAC-3 System 
was approved and PAC-3 system was fielded to the Army. This was due to 
substantial increase in system performance in the IOTE over that demonstrated in 
the LUT and there was valuable benefit to the Army. Current operations in the 
mid-east seem to validate these decisions. 

(4) Operational Suitability Issue, page 32, Reliability paragraph. Non-concur. 
Rationale: The draft DOD IG report states the LUT revealed a number of PAC-3 
system reliability problems: (1) The communication relay group Mean Time 
Between Critical Mission Failure (MTBCMF) was 14 percent of the allocated value 
(the allocated value is a subset of the requirement); (2) the engagement control 
station MTBCMF was approximately half the allocated rate; and (3) the fire unit 
MBCMF was 60 percent of the allocation. The Routing Logic Radio Interface Unit 
(RLRIU) problems were the primary reasons for these higher numbers. It should 
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Final Report 


Reference 
4 
be noted that the Lower Tier Project Office (LTPO) and Raytheon resolved the 
RLRIU problems after the FUE-G LUT and prior to the FUE-M IOTE. This 
correction is indicated by the improved MTBCMF from 12.6 hours during LUT to 
19.4 hours during IOT. 
c. Section Ill - TACTICAL UNMANNED AERIAL VEHICLE (TUAV). 

, . . Revised 
(1) Operational Effectiveness Issues, pages 32-33, Ground Control Station and Page 40 
TCS compatibility paragraph. Concur with this section as written. § 
(2) Operational Suitability Issues, pages 10 and 33,”Logistics Supportability” 
paragraphs. Non-concur. Recommend deletion of both paragraphs within the Page 10 
document. Rationale: As defined by paragraph 4.a.(6), “Required Capabilities” of Page 40 


the TUAV Operational Requirements Document (ORD), threshold fuel 
requirements for the TUAV system were met by both the TUAV system's air 
vehicles and generators at the May 2002 Initial Operational Test (IOT). Although 
a system limitation memorandum was approved for the TUAV IOT, it references 
an inconsistent program support requirement within paragraph 5a of the ORD and 
should be disregarded as input to the subject report. Per the current TUAV Test 
and Evaluation Master Plan, ATEC will conduct a Limited User Test to reevaluate 
the system upon development and integration of a replacement heavy fuel 
engine. 
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Department of the Navy Comments 


DEPARTMENT OF THE NAVY 
OFFICE OF THE ASSISTANT SECRETARY 
RESEARCH, DEVELOPMENT AND ACQUISITION 
1000 NAVY PENTAGON 
WASHINGTON, DC 20350-1000 


23 April 2003 


GENERAL FOR AUDITING 


FOR DEFENSE SYSTEMS", REPORT D-2001AE-0123 


Ref: (a) DODIG Draft Audit Report D-2001AE-0123 


Encl: (1) Navy Comments Per OPNAV N91 ltr of 15 Apr 03 


with the report and outlines our position on all 
recommendations. 


information. 


Michael McGrath 
Deputy Assistant Secretary 
of the Navy 


Evaluation 
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MEMORANDUM FOR THE DEPARTMENT OF DEFENSE ASSISTANT INSPECTOR 


Subj: DEPARTMENT OF THE NAVY COMMENTS ON DODIG DRAFT REPORT ON 
"MANAGEMENT OF DEVELOPMENTAL AND OPERATIONAL TEST WAIVERS 


ASN(RDA) appreciates the opportunity to review reference 
(a) and offer comments to the draft report. Upon review of the 
report, Navy nonconcurs with the recommendations of the draft 
report. Enclosure (1) summarizes the areas of discrepancies 


The Navy point of contact for this audit is CDR Rick 
Scudder, N912, at (703) 601-1731 should you need additional 


Research, Development, Test & 


Deliberative process privilege data omitted. 
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Deliberative process privilege data omitted. 
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Deliberative process privilege data omitted. 
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Deliberative process privilege data omitted. 
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Deliberative process privilege data omitted. 
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Deliberative process privilege data omitted. 
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Deliberative process privilege data omitted. 


104 


Deliberative process privilege data omitted. 


105 


Deliberative process privilege data omitted. 
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Deliberative process privilege data omitted. 
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Department of the Air Force Comments 


Final Report 
Reference 
DEPARTMENT OF THE AIR FORCE 
HEADQUARTERS UNITED STATES AIR FORCE 
Washington, DC 
17 APR 2003 
MEMORANDUM FOR DOD IG 
FROM: HQ USAF/TE 
1650 Air Force Pentagon 
Washington DC 20330-1650 
SUBJECT: Report and Comment on the Management of Developmental and Operational Test 
Waivers for Defense Systems (Project No. D2001 AE-0123) 
AF/TE reviewed and provided comment on an earlier version of the subject report, 
Attachment 1. 
The current request for review and comment of the subject report has been completed with the 
finding that very few of our original comments have been included: 
a. Pages 15 through 27, missing in the original report, have been included in the current Pages 24 
version of the report. through 34 
b, Comments on page 9 of the original report referencing Joint Direct Attack Munition 
(JDAM) minor problems have been removed. Deleted 


c. Reference to Air force Operational Test and Evaluation Center Instruction 99-102 has 
been corrected. 

d. Comments regarding F-15 TEWS operational suitability issues have been included on 
page 40. 

e. Comments regarding JDAM operational suitability issues have been removed. 


Since the majority of our original comments have not been addressed, not been included, and 
apparently not considered for inclusion in the current version of the report, our original comments 
(Attachment 1) still stand as a valid submission. 


In regard to the report’s recommendation to update guidance for waivers and test deferrals, 
AF/TE is in the process of rewriting its AFIs and will replace AFI 99-101 and AFI 99-102. 


AF/TE concurs with AFOTEC’s comments (Attachment 2) on the subject report and is 
provided for your review. 
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The point of contact in AF/TE is Mr. Frank Koval, (703) 697-0296, e-mail 
Frank.koval@Pentagon.af.mil. 


“4A 
7 
? 


an bilan 


JOHN T. MANCLARK 
Director, Test and Evaluation 


Attachments: 
1. HQ USAF/TE Comments, 9 Jan 03 
2. AFOTEC Comments, 24 Mar 03 


ec: 

SAF/AQX 

HQ AFOTEC/XP 
HQ AFMC/DOX 
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DEPARTMENT OF THE AIR FORCE 
HEADQUARTERS UNITED STATES AIR FORCE 


9 Jan 03 


MEMORANDUM FOR Department of Defense Inspector General (DoD IG) 
(ATTN: JACK SNIDER) 


FROM: HQ USAF/TE 
1650 Air Force Pentagon 
Washington DC 20330-1650 


SUBJECT: Review and Comment on Preliminary Draft Report, “Management of Developmental 
and Operational Test Waivers for Defense Systems” 


Thank you for forwarding an early draft of the subject report. We have a number of major 
concerns with the document and have attached a point paper with more details, 


The August 3, 2001, memo referenced in the report did not reach HQ USAF/TE or the Air 
Force Operational Test and Evaluation Center (AFOTEC). A different memo dated 7 August, 2001 
was sent to AFOTEC, however, it did not contain any guidance about test and evaluation (T&E) 
waivers. We have since obtained a copy of the August 3, 200] memo addressed only to the 
DOT&E Staff. Had the Air Force been given the opportunity to review or coordinate on the August 
3, 2001 memo, we would have non-concurred with the language about waivers and requirements. 
We are unsure if the August 3, 2001 memo was fully disseminated to the entire TRE community. 


The draft report reveals great differences in opinion, policy, and practice among the Services 
and DOT&E staff about what constitutes a “waiver.” The Air Force believes the draft report 
incorrectly considers test waivers, test limitations, and test deferrals as the same thing. The Air 
Force rarely applies outright for test waivers, but identifies test limitations and test deferrals to 
DOT&E through the test plan approval process. Once DOT&E reviews and approves the test plans, 
there is no need for additional action. We are following DoD 5000 policy and Title 10 which do not 
require waivers for test limitations or test deferrals. Nonetheless, we are currently revising all HQ 
USAF T&E policy documents and plan to address the use of waivers in more detail. The other 
Services appear to use different strategies and policies. 


The draft report fails to recognize that USD(AT&L) institutionalized evolutionary acquisition 
(EA) in the latest DoD 5000-series as the “preferred acquisition strategy.” The draft report also fails 
to recognize the principles of time-phased requirements articulated in CJCSI 3170.01B, 
Requirements Generation System. The main point is that deferrals of some requirements and 
capabilities, and therefore testing of the associated critical operational issues and Tequirements, must 
now be planned for so that EA and time-phased requirements will work. Materiel developers, 
testers, and users can no longer expect a technically perfect solution or “the perfect test” in the first 
increment that’s deployed. USD(AT&L) mandated that trade-offs be made between cost, schedule, 
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and performance so development programs get interim capabilities delivered sooner while keeping 
cost and schedule under control. 


The three Air Force programs cited (Predator, Joint Direct Attack Munition, and F-15 Tactical 
Electronic Warfare System) all used the EA strategy and time-phased requirements to varying 
degrees to field new capabilities earlier than the traditional acquisition process. The examples of 
test waivers, limitations, and deferrals cited in the report do not rise to a level of significance that 
warrant delaying any of these programs. Program managers are following DoD 50000 guidance to 
field capabilities to warfighters sooner. 


Finally, Air Force testers were not in-briefed or consulted during this audit. Since the draft 
report discusses Service development programs and T&E policies, the DoD IG should conduct 
interviews with Air Force and other Service officials in charge of these areas. To that end, we look 
forward to discussing the details of the draft report and the attached paper with you at your earliest 
convenience. The points of contact in HQ USAF/TE are Lt Col Bo Tye, (703) 697-0190, e-mail 
Bo. Tye@Pentagon.af.mil, and Mr. Chuck Triska, (703) 697-0299, e-mail 


Charles. Triska@Pentagon.af.mil. 
#f signed # 
DAVID HAMILTON 
Deputy Director, Test and Evaluation 
Attachment: 


Discussion Paper 


cc: 

SAF/AQK 

HQ AFOTEC/XP 
HO AFMC/DOX 
412 TWICC 
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Discussion Paper 
on 
Draft Proposed DoD IG Report 
on 
“Management of Developmental and Operational Test Waivers for Defense 
Systems” 


OVERVIEW: This paper comments on the subject draft DoD IG report. Information in the draft 
appears to come solely from the Director, Operational Test and Evaluation (DOT&E) with minimal 
to no input from the Services. There was no in-brief or interviews with Air Force testers that we 
know of. 


DISCUSSION: 


- Page 1, paragraph 1, lines 7-11: We agree that “DoD guidance does not specifically define what a 
waiver from developmental and operational testing is.” There is no officially coordinated definition 
or policy. The only DoD-level test and evaluation (T&E) waivers we know of descend from Title 
10 §2399 and §2366 with regards to system contractor involvement in operational test and 
evaluation (OT&E) and when live fire test and evaluation (LFT&E) is too expensive and 
impractical. The Air Force recognizes and uses these kinds of waivers when necessary. 

-- However, we disagree with DOT&E’s definition of a waiver as a “deferral of a testing 
requirement, which does not eliminate the requirement for testing.” Deferring the test of a system 
requirement to a later point in program development does not mean the need to test it is waived. 
This definition is not officially recognized DoD-wide, nor was it coordinated with the Service 
testers. It is evident the concept of a “test waiver” varies greatly among the Services. 

-- In addition to the Title 10 waivers, Air Force Instruction (AFI) 99-102, Operational Test 
and Evaluation, requires “waivers” in three other instances: 

--- Program managers (PM) must obtain a waiver from Air Force Materiel 
Command to use non-DoD test facilities when such facilities are available (paragraph 3.10.2); 

--- AFOTEC must obtain a waiver from HQ USAF/TE before AFOTEC may 
transfer responsibility for an OT&E program to a major command (paragraph 6.23.2); and 

--- AFOTEC must obtain a waiver from HQ USAF/TE before AFOTEC can alter, 
exclude areas from, or not conduct an OT&E (paragraph 2.5). AFOTEC needs user and developer 
concurrence before HQ USAF/TE can process these kinds of waivers. 

--- In addition, AFMAN 63-119, Certification of System Readiness for Dedicated 
OT&E, paragraph 4.2.7, states, “Approval of a waiver does not eliminate or alter the requirement 
for OTXE. Waived items must be tested in subsequent OT&E or the ORD must be changed.” 
These are waivers that result from the process for Certification of Readiness for OTRE. 


- Page 1, paragraph 2, line 3: We know of no requirement for the Services to obtain waivers for 
DT&E of any kind. In addition, DOT&E is restricted by Title 10 §139(d) as follows: “The 
Director may not be assigned any responsibility for developmental test and evaluation, other than 
the provision of advice to officials responsible for such testing.” 


- Page 2, paragraph 1, line 5-6: The report alleges there are “waiver equivalents” called “test 
limitations” and “[test] deferrals.” These are not the same as a waiver, The report's language 
genetates confusion about what is a waiver and when it’s required. 
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Page 1 


Page 1 


Pages 1 
through 2 


Final Report 


Reference 


Revised 
Page 3 


Page 3 


Page 3 


Page 4 


-- The Air Force rarely applies outright for test waivers, but instead identifies test 
limitations and test deferrals to DOT&E through the test plan approval process. Once DOT&E 
reviews and approves the test plans, to include the test and evaluation master plan (TEMP), there is 
no need for additional action. We follow DoD 5000 and Title 10 which do not require waivers for 
test limitations or test deferrals. Submission of a separate waiver request in addition to an approved 
test plan is a redundant action. 

~~ Test limitations are not test waivers. They are constraints to an ideal test strategy 
or plan imposed by circumstances largely beyond everyone’s control. They are documented in test 
plans and TEMPs along with ways to mitigate the impacts and still produce acceptable evaluations. 

--- Test deferrals are not test waivers. They are agreements that certain parts of a 
T&E program may be delayed until a later point in systems development. They are based on trade- 
off decisions mutually acceptable to developers, testers, and users, and are documented in test plans 
and TEMPs. 


- Page 2, paragraph 2, line 2: Multiple comments: 

-- First, there is no requirement to resolve all critical operational issues (COI) before 
proceeding into full-rate production (FRP) or deployment for software-intensive systems. COIs are 
tester-developed and not operational requirements. Some COIs such as operational suitability 
require a much greater amount of testing so that the operational requirement may not be satisfied 
until later. The new reformed acquisition environment allows the milestone decision authority 
(MDA) to use trade-off analysis to defer the closure of COIs when it makes good sense for the 
program. 

-- Second, the fact that some COIs are not resolved prior te FRP or deployment is not 
caused solely by the testers (DT&E and OT&E), but is the final responsibility of the materiel 
developer and the MDA. The tester’s job is to provide information to the developer and MDA 
within the constraints of the acquisition and T&E strategies. The MDA makes the final decision to 
proceed into production (not the testers) based on the COIs there were resolved, taking into account 
those COls that were not resolved. 


- Page 2, paragraph 3, all: Appendix C is missing along with pages 15 through 27. DOT&E is not 
qualified to speak for the Services regarding Service T&E policies. DoD IG should come to the 
Service testers to complete this section. 


~ Page 2, paragraph 4, lines 6-11: We note some problems with the August 3, 2001, DOT&E 
memo and its attachment. First, the memo was not addressed to the Services, but only to the 
“DOT&E Staff,” and was not coordinated with the Services before publication. It included a few 
sentences about test waivers with which we fundamentally disagree such as, “There will be no 
waivers, or deferrals, to the completeness of evaluations. Requirements waived should be 
considered failed unless changed.” If testing a requirement is deferred because a support asset was 
not available during test, this does not mean the system “failed” to meet the requirement. These 
statements are toc extreme and run counter to the spirit of streamlined acquisition in the current 
DoD 5000-series and CJCSI 3170.01B, Requirements Generation System. 


- Page 3, paragraph 4, Air Force Policy, last line: A key caveat overlooked in this paragraph (and 
the rest of the report) are the last two words, “unless unavoidable.” These words allow for 
unforeseen circumstances in development programs and T&E strategies that would do great harm to 
programs if no management flexibility were allowed. In some instances, it is better to defer a test 
requirement or completion of a COI than severely penalize the program with cost and schedule 
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impacts resulting from a less than perfect development or T&E program. This caveat allows PMs, 
testers, and users to make reasonable trade-offs in the best interests of warfighters according to DoD 
5000’s new streamlined acquisition principles. 


- Page 3, paragraph 5, multiple comments: 

-- Occasionally, some COIs are not possible to resolve in initial OT&E (IOT&E) prior to 
FRP or deployment due to the nature of the ORD requirement and/or test limitations. Consider a 
suitability requirement for 1000 hours mean time between failure for an aircraft system. 
Demonstrating this requirement may require up to 5,000 flying hours in some instances. It may not 
be possible or good program management to insist that 5,000 flying hours are logged before 
making a FRP or deployment decision. Other tools such as teliability growth curves can reduce risk 
and help make good projections when these kinds of test limitations occur. 

~ The draft report again confuses test waivers with test limitations and test deferrals. This 
problem continues throughout the report. 


- Page 4, Table of COls Not Fully Resolved: The table is out of balance because it’s focused 
almost solely on suitability COIs. As explained in the previous paragraph, some suitability 
requirements and COIs are notoriously hard to achieve in the limited time frame of a development 
program and an OT&E. Trying to achieve 5,000 flying hours was an example of a suitability 
requirement or COI that is a test limitation, and which may require deferral of a final answer that 
resolves the requirement or COI until a later date. EA strategies recognize this may happen. 


- Page 5, paragraph 4, lines 1-4: The draft states that the Joint Requirements Oversight Council 
(GIROC) deferred some test requirements as part of an evolutionary development program. This is a 
good example of how evolutionary acquisition (EA) works to ensure earlier delivery of capabilities 
to warfighters. Evolutionary acquisition is the “preferred acquisition strategy” in DoD. It is also 
not justified to criticize the Patriot program for deferring a requirement after careful deliberation 
with the JROC and deploying much-needed capabilities to protect our warfighters. 


- The next 6 pages (5 through 10) cite 11 case studies focused on relatively minor problems that did 
not warrant delaying the FRP or deployment decisions. There are no perfect development programs 
and no perfect T&E programs. Waiting for every problem to be resolved, no matter how small, 
before FRP or deployment would result in no systems being deployed, or at best would add totally 
unacceptable cost and delay to programs. The decision to “waive” test requirements or defer some 
tests is a mutual agreement between the testers, materiel developers, and users based on program 
and technical risk weighed against the risk of not deploying new capabilities to our warfighters. 
Sometimes “better” is the worst enemy of “good enough.” 


~ Page 6, paragraph 3: The draft criticizes the Predator for not achieving its full potential before 
FRP and deployment. It was intended as a fast-track program from the start and follows the new 
DeD 5000-series EA acquisition model. It is not justified to criticize Predator solely on technical 
and testing grounds since Predator delivered new capabilities to warfighters earlier just as the new 
DoD 5000-series envisions. DOT&E’s criticism of Predator has little credibility because top-level 
combatant commanders had the highest praises for Predator despite its drawbacks. Note that 
DOT&E’s latest draft Annual Report to Congress has withdrawn last year’s conclusion that 
Predator was not effective and not snitable. 
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Page 8 


Deleted 


Page 12 


Page 14 


Page 17 


Page 17 


- Page 7, paragraph 3: The draft criticizes the F-15 Tactical Electronic Warfare System (TEWS) 
test plans for not showing test limitations. The deficiency was known to DOT&E at the time, so no 
waivers were requested or submitted, and DOT&E approved the test plans as submitted without 
comment in this area. Test plan approval covered the test limitations and test deferrals. DOT&E 
knew that the Air Force would attempt to correct the deficiency in the next incremental release 
which is how EA is supposed to work. 


- Page 9, paragraph 1, multiple comments on the Joint Direct Attach Munition (JDAM): 

-- First, HQ USAF/TE did not require approval of waivers or test deferrals of any kind for 
this program. While we recognize that some tests are deferred, HQ USAF/TE keeps close watch on 
these matters when reviewing OT&E plans before submission to DOT&E. 

-- Second, AFOTEC decertified the Joint Programmable Fuse (JPF) for OT&E, not the 
JDAM. This test limitation, outside the control of the OAM PM, should not hold the JDAM 
program hostage. Other fuses were available that afforded adequate capabilities to demonstrate 
JDAM. Fuse delays are only a small part of the total weapons effects equation, so the total impact 
to warfighters of not having JPF is minimal. Z 

-- Third, JDAM’s re-targeting capabilities are not affected by lack of IPF. Re-targeting is 
an capability inherent in JDAM itself and not dependant on the fuse. The only problem is if the 
IDAM were retargeted in flight, the fuse setting may not always be precisely compatible with the 
new target. For troops in contact who need ordnance right away, this criticism does not pass the “so 
what” test. 


- Page 10, paragraph A.1: Non-concur. The words “waivers or their equivalents” are not 
uniformly accepted terms or practices across DoD and the Services. Therefore, that terminology is 
too problematic as discussed earlier in this paper. The August 3, 2001, memo was not coordinated 
with the Air Force, and we have fundamental disagreements with its statements on waivers and 
requirements. 

~ Page 10, paragraph A.2: Non-concur. The recommendation is counter-productive and 
undermines USD(AT&L)’s new direction in the DoD 5000-series to make EA the preferred 
acquisition strategy. MDAs must retain sufficient management flexibility to plan wisely, and must 
retain authority to make needed trade-offs between cost, schedule, and performance for the good of 
the program. If implemented, large cost overruns and long delays to fielding new capabilities to 
warfighters would become routine. 


- Page 11, paragraph 1, lines 9-13: We have not been issued this DOT&E direction, nor were we 
afforded a chance to coordinate. This matter was discussed earlier. 


Page 11, paragraph 2, 3" bullet: Multiple comments: 

-- First, we do not agree that “Air Force procedures require the processing of waivers to 
defer specific operational test requirements to the future.” While Air Force test policies do address 
waivers, those policies are not so specific. As explained earlier, the Air Force addresses test 
limitations and test deferrals through the test plan approval process. During this process, our 
intention is that AFMAN 63-119, paragraph 4.2.7, be followed, namely, “Approval of a waiver does 
not eliminate or alter the requirement for OT&E. Waived items must be tested in subsequent 
OT&E or the ORD must be changed.” We accomplish the same goal in the end. 

-- Second, the Air Force has no HQ USAF-level policies addressing procedures for PMs to 
use test limitations in test plans to defer specific operational test requirements to the future. We 
know of no major command policies in this area, either. 
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-- Third, Air Force T&E policies are under revision, A new AFI 99-103, Air Force Test 
and Evaluation, is already in a mature stage of development and will address this area. 


- Page 13, paragraph 2, Air Force Procedures: Multiple comments: 

-- We have not published or heard of a January 20, 2000 version of AFI 99-102. Could this 
possibly be AFOTEC Instruction 99-102? We did not approve this document before publication, 
and have net reviewed it in conjunction with our review of your draft report, 

— AFMAN 63-119, not AFI 99-102, requires Air Force PMs to “list any required waivers or 
areas excluded from OT&E” during the certification process and in the certification message. It 
also says “approval of'a waiver does not eliminate or alter the requirement for OTXE. Waived 
items must be tested in subsequent OT&E or the ORD must be changed.” 

-- We could not locate where this statement was made in HQ USAF policy instructions: 
“identify test limitations in test plans to defer the demonstration of specific operational test 
requirements...” Could this be in AFOTEC Instruction 99-1027 

++ Although AFMAN 63-119 uses the term “waiver,” HQ USAF/TE does not require 
AFOTEC to submit formal paperwork for “acceptance” of an OT&E with test limitations or 
“waivers.” We do not require waivers for missing DT&E areas, We leave that between the 
AFOTEC commander and the program’s certifying official (usually the PEO). The AFOTEC 
commander makes the final decision, but must still get his test plan coordinated through HQ 
USAF/TE and approved by DOT&E before starting the OT&E. 


- Page 14, first paragraph: We agree with the conclusion that the Services do not have consistent 
policies, procedures, or terminology for test waivers and limitations. 


- Page 14, paragraph 2, multiple comments: 

-- In lines 7-8, the words “initial operational test and evaluation” read “FOT&E” in the 
latest version of DoDI 5000.2, paragraph 3.7.6, pending signature on January 13, 2003. That 
change is very significant. 

-- In lines 8-14, The Air Force did not review or coordinate on the August 3, 2001, memo 
and would have non-concurred with language on waivers and requirements. We do not know what 
the other Services received. We are unsure if the August 3, 2001, memo was fully disseminated to 
the Service T&E communities. 


- Page 14, Recommendation B: Partly concur. We agree that uniform policies about waivers may 
possibly be good policy for the DoD, but we non-concur with making all Service polices, and 
particularly Air Force policies, mirror DOT&E’s policies. DOT&E should first open a dialogue 
with the Services and seek consensus before institutionalizing new policies, Imposition of a “one- 
size-fits-all!” approach could undermine USD(AT&L)’s attempts to streamline acquisition using EA 
and time-phased requirements in the new DoD 5000-series. 


- Pages 15 to 27 are missing. These pages include Appendices C and D. 


- Page 38, F-15 TEWS Operational Suitability Issues: The draft report fails to mention that 
DOT&E approved the Band 1.5 TEMP and OTE plan, both of which addressed the false alarm 
rate and future testing that was needed. The draft report also fails to mention that the Air Force 


submitted plans to DOT&E to remedy the false alarm rate problems in the next increment (Band 
5.0). 
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- Page 39, Joint Direct Attack Munition: Most of the issues in this paragraph were addressed 
previously. In addition, note that the JPF system program office (SPO), not the JDAM SPO, is 
responsible for finding solutions to this interoperability problem. Also note that FOT&E is a 
natural follow-up to IOT&E whether or not the JPF was included in the IOT&E. FOT&E was not 
planned solely because JPF was not available. 


C. Triska, HQ USAF/TEP (SAIC), 9 Jan 03, (703) 697-0299 
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DEPARTMENT OF THE AIR FORCE 
HEADQUARTERS AIR FORCE OPERATIONAL TEST AND EVALUATION CENTER 
KIRTLAND AIR FORCE SaSE, NEW MEXICO 


24 Mar 03 


MEMORANDUM FOR USAF/TE 


FROM: AFOTEC/CV 
8500 Gibson Blvd SE 
Kirtland AFB NM 87117-5558 


SUBJECT: DoD IG Draft Report, Management of Development and-Operational Test Waivers 
for Defense Systems, (Project Code D2001AE-0123) 


1. This is in reply to your tasker requesting AFOTEC provide comments to the subject draft 
report. 


2. AFOTEC does not concur with all the conclusions reached in this feport. Our concerns fall in 
to two primary areas; 1) A flawed assumption on the part of the DoD IG evaluators regarding 
the defintion of a “waiver” and 2) assuming a DOT&E internal memorandum is a policy directive 
to the OTAs. Detailed comments on the draft report are provided in the attachment. 


3. My POC for this action is Lt Col Kevin Grill, XPY, DSN 246-9507 and E-mail 


kevin.g il@afotec.a mil 


Colonel, USAF 
Vice Commander 


Attachment 
AFOTEC Comments to DoD IG Draft Report 
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AFOTEC Comments to DoD IG Draft Report, 
Management of Developmental and Operational Test Waivers for Defense 
Systems, (Project Code D2001AE-0123) 


4. General Comment: The DoD {G's assumption that test waivers, limitations, and deferrals 
are equivalent terms is not an acceptable assumption to AFOTEC. Each of these terms has 
their own specific meaning and use. This assumption creates a misinterpretation of facts and 
data throughout the rest of the report. 


2. Page 3, “DoD Policy”, first paragraph 

Comment: DoD JG cites the Director of OT&E Memorandum, dated 3 Aug 2001, as the DoD 
level policy for directing service OTAs to not use waivers or deferrals that prevent a complete 
evaluation of operational requirements. AFOTEC does not consider this memo as policy, but 
rather guidance to Mr. Christie's staff to ensure the goal and objective of “rigorous and robust 
testing that is adequate" is implemented. This same guidance was provided as a courtesy to 
the AFOTEC Commander in 4 subsequent letter, dated 7 Aug 2001. DOT&E policy is officially 
promulgated through DOT&E Policy Letters. AFOTEC immediately implements these policy 
letters upon receipt through updates to AFOTEC Directives. 


3. Page 13, “B. Waiver Policies and Procedures” 

Comment: AFOTEC does not implement a policy to waive or defer operational test 
requirements. The AFOTEC Commander supports Mr. Christie's goal of rigorous, robust and 
adequate testing, without waivers or deferrals, by implementing the policy outlined in AFOTEC 
jnstruction 99-103, “Conduct of OT&E”, paragraph 4.7. The AFOTEG Commander will not 
accept certification of a system readiness for OT&E until the developer can demonstrate 
stabilized performance under an operational (stressed) environment with a production 
representative article. AFOTEC Test Teams are advised that AFMAN 63-119, “Certification of 
System Readiness for Dedicated OT&E”, tempiates 8, 15 and 19 are critical te ensuring, in 
collaboration with the System Program Office, the system can be successfully certified. 


4, Page 15, “Air Force Procedures, Test and Evaluation Waivers” 

Comment: Air Force Instruction 99-102, “Operational Test and Evaluation’, is referred to by the 
DoD IG when defining the Air Force's policy on test and evaluation waivers. Paragraph 2.5 is 
quoted, and AFOTEC believes incorrectly interpreted by DoD IG. AFOTEC refers to this 
paragraph when submitting a waiver to the requirement to do OT&E, not to waive or defer 
testing of operational requirements. The DoD IG report states that the system program 
manager is the individual that submits weivers under AFI 99-102, paragraph 2.5. This is not a 
correct statement. The AFOTEC program management submits the waiver request, with user 


and developer concurrence, through the AFOTEC/CY, who in-turn submits the request to 
AF/TE. 


5. Page 23, “Waiver” 

Comment: The DoD IG report contains DOT&E’s definition of a waiver. AFOTEC has 
searched Title 10, past versions of DoD 5000 series directives, the current OSD acquisition 
interim directives and guidance, and the current draft DoD 5000 series directives for DOT&E’s 
waiver definition. We cannot find this definition. Nor does policy or guidance exist in any of 
these documents on the process to waive or defer, or not waive/defer, testing of operational 
requirements. AFOTEC recommends that if DOT&E desires to establish a policy not allowing 
the wavier or deferral of testing operational requirements, such a statement neads to be 
documented in the current draft version of the DoD 5000 series directives. 
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6. Page 17, Appendix A. “Scope and Methodology”, 7th bullet Page 24 
Comment: Nowhere does it identify what conclusion the 1G team came to in determining the 
role of Institute for Defense Analysis (IDA) in the T&E waiver process. it is the opinion of 
AFOTEC that IDA has NO role in the process. They are simply a support contractor to DOT&E. 
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